3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is...
Transcript of 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is...
![Page 1: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/1.jpg)
3D Shape Regression for Real-‐3me Facial Anima3on
Chen Cao, Yanlin Weng, Stephen Lin, Kun Zhou
FaceWarehouse: a 3D facial Expression Database for Visual Compu3ng
Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, Kun Zhou
Zhejiang University MSRA
Presented by Shu Liang (Black-‐on-‐white slides are Shu’s)
![Page 2: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/2.jpg)
Facial Animation
• Facial animation is widely used in films & games • Performance-based facial animation
Avatar 2009 © 21st Century Fox
L.A. Noire 2011 © Team Bondi
![Page 3: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/3.jpg)
Related Work
• Performance-based Facial Animation
Special Equipment Camera arrays
[Beeler et al. 2011]
Structured light [Weise et al. 2009]
Quality
Facial Markers [Huang et al. 2011] Device complexity
![Page 4: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/4.jpg)
Related Work
• Performance-based Facial Animation
Single Camera ASM & AAM [Cootes et al. 1992-2001]
Regression-based alignment [Cao et al. 2012]
Quality
Optical Flow [Vlasic et al. 2005] Device
complexity
![Page 5: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/5.jpg)
[Bouaziz et al. 2013] [Li et al. 2013]
Related Work
• Performance-based Facial Animation Consumer RGBD Camera Quality [Weise et al. 2011]
Device complexity
![Page 6: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/6.jpg)
Our Goal
Quality
Device complexity
• Real-time facial animation for average users
![Page 7: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/7.jpg)
Our Goal • Real-time facial animation for ordinary users
– Single web camera – Robust
• Fast motions • Large rotations • Exaggerated expressions
– General environments • Indoors and outdoors
– High performance • Mobile devices
![Page 8: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/8.jpg)
Our Pipeline
![Page 9: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/9.jpg)
…
Our Pipeline
• One-time Preprocess
with labeled 2D landmarks
… Captured images
User-specific Blendshapes
3D Shape Regression Training
User-specific 3D Shape Regressor
… 3D Facial Expression Database
Training Data Construction
… …
Training Images and Shapes
![Page 10: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/10.jpg)
… User-specific Blendshapes
Our Pipeline
• Runtime computation
User-specific 3D Shape Regressor
3D Shape Regression Tracking Animation
Video Stream Tracked Mesh Digital Avatar 3D Facial Shape
![Page 11: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/11.jpg)
3D Face Shape Regression: Preprocess • Data Collection
– Image capturing & labeling – Blendshapes generation – Shape reconstruction – Training data generation
• Training
![Page 12: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/12.jpg)
Our Pipeline
![Page 13: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/13.jpg)
Preprocess: Image Capturing & Labeling
…
Captured Images
…
[Cao et al. 2012] + Manual Adjustment Labeled 2D Feature Points
![Page 14: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/14.jpg)
Our Pipeline
![Page 15: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/15.jpg)
…
User-specific Blendshapes
Fitting … …
Preprocess: Blendshapes Generation
…
Labeled 2D Feature Points …
… …
FaceWarehouse [Cao et al. 2013]
150 identities × 47 expressions
![Page 16: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/16.jpg)
Our Pipeline FaceWarehouse
![Page 17: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/17.jpg)
FaceWarehouse - RGBD images of 150 individuals captured by Kinect - Aged 7-80 from various ethnic backgrounds - Different expressions, one neutral and 19 other expressions.
17
![Page 18: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/18.jpg)
- Mesh deforma3on Neutral expression
Other expressions
S0, S1, S2 … S19 for 20 expressions.
FaceWarehouse
18
[Blanz et. al 2004] [Huang et. al 2006]
[Sumner et. al 2006] [Huang et. al 2006]
![Page 19: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/19.jpg)
- Individual-‐specific expression blendshapes • Example-‐based facial rigging algorithm: An expression H of the person can be:
46 FACS blendshapes
• Begins with a generic blendshape model • Op3mized by minimizing the difference between Sj and linear
combina3on of Bi with known weight for expression j, the difference between the deforma3on from B0 to Bi and that from A0 to Ai.
FaceWarehouse
19
{B1,B2,…B46}
![Page 20: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/20.jpg)
- Bilinear face model
A rank-‐three data tensor T. (11K ver3ces × 150 iden33es × 47 expressions) Used N-‐mode SVD to decompose the tensor.
FaceWarehouse
20
![Page 21: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/21.jpg)
Our Pipeline
![Page 22: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/22.jpg)
Image Mesh Shape
… … …
Preprocess: 3D Shape Reconstruction
![Page 23: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/23.jpg)
…
… … … … …
𝐼𝑖,𝐌𝑚,𝑆𝑖𝑚 𝑎
𝐌𝑗𝑎,1 ≤ 𝑗 ≤ 𝑚 ≤ 𝑚
𝐼𝑖 𝑆𝑖𝑜
Preprocess: Training Data Generation • Data Augmentation
𝐼𝑖,𝐌1𝑎,𝑆𝑖1
Translations
𝐼𝑖,𝐌2𝑎,𝑆𝑖2
3D Shape Space
… …
![Page 24: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/24.jpg)
… … … … …
…
• Training Set Construction
𝐼𝑖,𝐌𝑗𝑎,𝑆𝑖𝑗,𝑆𝑖1𝑗1
𝐼𝑖,𝐌𝑗𝑎,𝑆𝑖𝑗
𝐼𝑖,𝐌𝑗𝑎,𝑆𝑖𝑗,𝑆𝑖𝐺𝑗𝐻
𝐼𝑖,𝐌𝑗𝑎,𝑆𝑖𝑗,𝑆𝑖1𝑗2 Find 𝐺 ⋅ 𝐻 guessed shapes ⋅ 𝐻 guessed shapes guessed shapes
3D Shape Space
… …
Preprocess: Training Data Generation Training Data
![Page 25: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/25.jpg)
Our Pipeline
![Page 26: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/26.jpg)
Preprocess: Training • Appearance vector • Primitive regressor: fern
• Summary: two-level boosted regressor
![Page 27: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/27.jpg)
…
𝐼𝑖,𝐌𝑖𝑎 𝑆𝑖𝑐 Appearance Vector
Preprocess: Training
• Appearance vector
![Page 28: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/28.jpg)
Clustering according to the feature vectors
…
…
𝛿𝑆𝑏1 𝛿𝑆𝑏2 𝛿𝑆𝑏3 𝛿𝑆𝑏𝐹
Preprocess: Training All Training Data
Randomly select feature vectors
• Primitive regressor
= …
𝛿𝑆𝑏 = = 1
1 + 𝛽/ Ω𝑏 𝑖∈Ω𝑏 𝑆𝑖 − 𝑆𝑖𝑐 − 𝑆𝑖𝑐
Ω𝑏
𝑆𝑖𝑐 = 𝑆𝑖𝑐 + 𝛿𝑆𝑏,𝑖 ∈ Ω𝑏 = 𝑆𝑖𝑐 + 𝛿𝑆𝑏,𝑖 ∈ Ω𝑏 + 𝛿𝑆𝑏,𝑖 ∈ Ω𝑏 ∈ Ω𝑏
![Page 29: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/29.jpg)
Calculate Appearance Vector …
𝛿𝑆𝑏1 𝛿𝑆𝑏2 𝛿𝑆𝑏3 𝛿𝑆𝑏𝐹
All training data Randomly select feature vectors Clustering according to the feature vectors …
Fern
Preprocess: Training
• Summary: two-level boosted regression Level One Level Two
![Page 30: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/30.jpg)
Our Pipeline
![Page 31: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/31.jpg)
3D Face Shape Regression: Runtime • Initialization: first frame • Following frames
– Find guessed shapes – Two-level boosted regression
![Page 32: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/32.jpg)
Runtime: Initialization
• First frame
Face Detection [Viola and Jones 2001]
2D Feature Alignment [Cao et al. 2012] 3D Shape Recovery
![Page 33: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/33.jpg)
… … … … …
Runtime: Following Frames
Previous Frame
• Find guessed shape Transform Current Frame
3D Shape Space … … Find similar shapes
Guessed Shape Transformed Shape
![Page 34: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/34.jpg)
Image & Guessed Shape Matched
Shape
Runtime: Following Frames
• Two-level boosted regression
Two-Level Boosted Regressor
![Page 35: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/35.jpg)
Our Pipeline
![Page 36: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/36.jpg)
Matched Shape Blendshape Weight
Rigid Transformation
Tracked Mesh Tracking
Digital Avatar Rigid Non-‐
Rigid
Tracking & Animation Similar to [Weise et al. 2011] …
User-specific Blendshapes
![Page 37: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/37.jpg)
Evaluation: Regressed shape vs. Kinect 3D vs. 2D vs. Optical Flow
37
![Page 38: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/38.jpg)
Live Demo • Demo
![Page 39: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/39.jpg)
Evaluation: Regressed shape vs. Kinect
![Page 40: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/40.jpg)
Evaluation: 3D vs. 2D vs. Optical Flow
![Page 41: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/41.jpg)
More Results: Outdoor
![Page 42: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/42.jpg)
Our System on Mobile Device
![Page 43: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/43.jpg)
Timings • Preprocess: 45 mins
– Capture: 10 mins – User interaction: 25 mins – Training: 10 mins
• Runtime: less than 15 ms – Regression: 5 ms – Tracking & Animation: 8 ms
![Page 44: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/44.jpg)
Limitations • Much training data
– 60 head poses and facial expressions
• Dramatic lighting changes
![Page 45: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/45.jpg)
Summary
• 3D facial performance capture from 2D video – – – –
Regression-based approach Robust: fast motions, large rotations, exaggerated expressions General environments: indoors and outdoors High performance: real-time
• Future work – Handle lighting variations – Reduce training data
![Page 46: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/46.jpg)
Acknowledgement • Face capture: Marion Blatt, Steffen Toborg • Anonymous reviewers • Funding
– NSFC (No.61003145 and No.61272305) – 973 program of China (No.2009CV320801)
• FaceWarehouse Data: http://gaps-zju.org/facewarehouse/
Thank you!
![Page 47: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/47.jpg)
Π𝐐 𝐌𝑖 𝐶𝑟 ×2 𝐰𝑖𝑑 𝐌𝑖 𝐶𝑟 ×2 𝐰𝑖𝑑 𝐶𝑟 ×2 𝐰𝑖𝑑 ×2 𝐰𝑖𝑑 ×3 𝐰𝑒𝑥𝑝,𝑖
Preprocess: Camera Calibration
𝑓 𝐐 = 0 = 0
0
0 𝑓 0
𝑢0 𝑣0 1
𝐸 = = 𝑛
𝑖=1 𝑘=1
75 𝑇 𝑇 𝑣𝑘
− 𝐮𝑖 𝑘
2 Blendshape Generations:
![Page 48: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/48.jpg)
Why not directly use previous shape? • Error accumulation
![Page 49: 3D#Shape#Regression#for#Real23me#Facial# Animaon ... · #Facial Animation # • Facial animation is widely used in films & games • Performance-based facial animation ... 2D landmarks](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eb1bc29c509f14f5d3aea15/html5/thumbnails/49.jpg)
Why not directly regress parameters? • Expression coefficients in [0:1] • Animation prior
– Temporal coherence
• Rigid transformation & expression coefficients – Different spaces