Post on 31-Mar-2018
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Face2Face:
Real-time Face Capture and Reenactment of
RGB-Videos
Justus Thies1, Michael Zollhöfer2, Marc Stamminger1,
Christian Theobalt2, Matthias Nießner3
1University of Erlangen-Nuremberg
2Max-Planck-Institute for Informatics
3Stanford University
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Related Work
• Offline • Online
RG
B-D
Real-time Expression Transfer for Facial Reenactment
Vdub: Modifying Face Video of Actors forPlausible Visual Alignment to a Dubbed Audio Track
Creating a Photoreal Digital Actor:The Digital Emily Project
RG
B
Face2Face: Real-time Face Capture and ReenactmentOf RGB-Videos
Spe
cial
Har
dw
are
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Related Work
• Offline • Online
RG
B-D
Real-time Expression Transfer for Facial Reenactment
Vdub: Modifying Face Video of Actors forPlausible Visual Alignment to a Dubbed Audio Track
Creating a Photoreal Digital Actor:The Digital Emily Project
RG
B
Face2Face: Real-time Face Capture and ReenactmentOf RGB-Videos
Spe
cial
Har
dw
are
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
• Face Capture• Energy Formulation
• Non-rigid Model-based Bundling
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
• Face Capture• Energy Formulation
• Non-rigid Model-based Bundling
• Reenactment• Mouth Retrieval
• Comparisons
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
• Face Capture• Energy Formulation
• Non-rigid Model-based Bundling
• Reenactment• Mouth Retrieval
• Comparisons
• Results / Live Demo
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Parametric Face Model
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6
𝑷 =
Φ𝛼𝛽𝛿𝛾
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6𝑷 = 6+80
𝑷 =
Φ𝛼𝛽𝛿𝛾
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6+80𝑷 = 6+80+80
𝑷 =
Φ𝛼𝛽𝛿𝛾
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6+80+80𝑷 = 6+80+80+76
𝑷 =
Φ𝛼𝛽𝛿𝛾
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 =
Φ𝛼𝛽𝛿𝛾
𝑷 = 6+80+80+76𝑷 = 6+80+80+76+27=269
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑃
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Face Capture
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
𝐸 𝑃 =
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
Distance inRGB Color Space
ColorConsistency
𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃
𝒍𝟐,𝟏 − 𝒏𝒐𝒓𝒎
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
Distance inImage Space
ColorConsistency
FeatureSimilarity
𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃 +𝐸𝑚𝑟𝑘 𝑃
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
RegularizationColorConsistency
FeatureSimilarity
𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃 +𝐸𝑚𝑟𝑘 𝑃 +𝐸𝑟𝑒𝑔(𝑃)
−𝟑 𝝈 +𝟑 𝝈𝟗𝟗, 𝟕%
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Non-rigid Model-based Bundling
𝐸𝑡𝑜𝑡𝑎𝑙 𝑷 =
𝑖=0
𝑛
𝐸𝑖 𝑷 → 𝑚𝑖𝑛
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Non-rigid Model-based Bundling
• Iterative Reweighted Least Squares (IRLS)
Gauss-Newton: 𝑱𝑻𝑱𝚫𝑷 = −𝑱𝑻𝑭
𝑱(𝑷) =
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Non-rigid Model-based Bundling
Inp
ut
Mo
de
l
Hierarchy Levels
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Tracking
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Tracking Comparison
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Tracking Comparison
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Tracking Comparison
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Reenactment
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
ReenactmentOnline RGB-Tracking
Preprocessed Video Tracking
Identity
Expression
Illumination
PoseP
er
Fram
e
Identity
Expression
Illumination
Pose
Pe
r Fr
ame
Reenactment
Expression Transfer
Mouth Retrieval
Compositing
Sou
rce
Act
or
Targ
et A
cto
r
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
ReenactmentOnline RGB-Tracking
Preprocessed Video Tracking
Identity
Expression
Illumination
PoseP
er
Fram
e
Identity
Expression
Illumination
Pose
Pe
r Fr
ame
Reenactment
Expression Transfer
Mouth Retrieval
Compositing
Sou
rce
Act
or
Targ
et A
cto
r
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Mouth-Retrieval
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Mouth-Retrieval
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Reenactment Comparison
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Live-Demo
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Limitations / Future Work
• Assumption of Lambertian surface and smooth illumination
• No occlusion handling
• No person specific details (fine scale details / wrinkles)
• Reenactment relies on a training sequence (Mouth retrieval)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Conclusion
• First Real-time Facial Reenactment only based on RGB-videos• Non-Rigid Model-Based Bundling
• Sub-Space Deformation Transfer
• Image-Based Mouth Synthesis
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Thank You!
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
References• O. Alexander, M. Rogers, W. Lambeth, M. Chiang, and P. Debevec.
The Digital Emily Project: photoreal facial modeling and animation.In ACM SIGGRAPH Courses, pages 12:1–12:15. ACM, 2009.
• P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Perez, and C. Theobalt.Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track.In Computer Graphics Forum. Wiley-Blackwell, 2015.
• F. Shi, H.-T. Wu, X. Tong, and J. Chai.Automatic acquisition of high-fidelity facial performances using monocular videos.ACM TOG, 33(6):222, 2014.
• C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou.Facewarehouse: A 3D facial expression database for visual computing. IEEE TVCG, 20(3):413–425, 2014.
• J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt.Real-time expression transfer for facial reenactment.ACM Transactions on Graphics (TOG),34(6), 2015.
• V. Blanz and T. Vetter.A morphable model for the synthesis of 3d faces.In Proc. SIGGRAPH, pages 187–194. ACM Press/Addison-Wesley Publishing Co., 1999.