Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference...
Transcript of Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference...
![Page 1: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/1.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Face2Face:
Real-time Face Capture and Reenactment of
RGB-Videos
Justus Thies1, Michael Zollhöfer2, Marc Stamminger1,
Christian Theobalt2, Matthias Nießner3
1University of Erlangen-Nuremberg
2Max-Planck-Institute for Informatics
3Stanford University
![Page 2: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/2.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Related Work
• Offline • Online
RG
B-D
Real-time Expression Transfer for Facial Reenactment
Vdub: Modifying Face Video of Actors forPlausible Visual Alignment to a Dubbed Audio Track
Creating a Photoreal Digital Actor:The Digital Emily Project
RG
B
Face2Face: Real-time Face Capture and ReenactmentOf RGB-Videos
Spe
cial
Har
dw
are
![Page 3: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/3.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Related Work
• Offline • Online
RG
B-D
Real-time Expression Transfer for Facial Reenactment
Vdub: Modifying Face Video of Actors forPlausible Visual Alignment to a Dubbed Audio Track
Creating a Photoreal Digital Actor:The Digital Emily Project
RG
B
Face2Face: Real-time Face Capture and ReenactmentOf RGB-Videos
Spe
cial
Har
dw
are
![Page 4: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/4.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
![Page 5: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/5.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
• Face Capture• Energy Formulation
• Non-rigid Model-based Bundling
![Page 6: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/6.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
• Face Capture• Energy Formulation
• Non-rigid Model-based Bundling
• Reenactment• Mouth Retrieval
• Comparisons
![Page 7: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/7.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Overview
• Parametric Face Model
• Face Capture• Energy Formulation
• Non-rigid Model-based Bundling
• Reenactment• Mouth Retrieval
• Comparisons
• Results / Live Demo
![Page 8: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/8.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Parametric Face Model
![Page 9: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/9.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷
![Page 10: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/10.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6
𝑷 =
Φ𝛼𝛽𝛿𝛾
![Page 11: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/11.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6𝑷 = 6+80
𝑷 =
Φ𝛼𝛽𝛿𝛾
![Page 12: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/12.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6+80𝑷 = 6+80+80
𝑷 =
Φ𝛼𝛽𝛿𝛾
![Page 13: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/13.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 = 6+80+80𝑷 = 6+80+80+76
𝑷 =
Φ𝛼𝛽𝛿𝛾
![Page 14: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/14.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑷 =
Φ𝛼𝛽𝛿𝛾
𝑷 = 6+80+80+76𝑷 = 6+80+80+76+27=269
![Page 15: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/15.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Parametric Face Model
𝑃
![Page 16: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/16.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Face Capture
![Page 17: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/17.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
𝐸 𝑃 =
![Page 18: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/18.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
Distance inRGB Color Space
ColorConsistency
𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃
𝒍𝟐,𝟏 − 𝒏𝒐𝒓𝒎
![Page 19: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/19.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
Distance inImage Space
ColorConsistency
FeatureSimilarity
𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃 +𝐸𝑚𝑟𝑘 𝑃
![Page 20: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/20.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Energy Formulation
RegularizationColorConsistency
FeatureSimilarity
𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃 +𝐸𝑚𝑟𝑘 𝑃 +𝐸𝑟𝑒𝑔(𝑃)
−𝟑 𝝈 +𝟑 𝝈𝟗𝟗, 𝟕%
![Page 21: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/21.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Non-rigid Model-based Bundling
𝐸𝑡𝑜𝑡𝑎𝑙 𝑷 =
𝑖=0
𝑛
𝐸𝑖 𝑷 → 𝑚𝑖𝑛
![Page 22: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/22.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Non-rigid Model-based Bundling
• Iterative Reweighted Least Squares (IRLS)
Gauss-Newton: 𝑱𝑻𝑱𝚫𝑷 = −𝑱𝑻𝑭
𝑱(𝑷) =
![Page 23: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/23.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Non-rigid Model-based Bundling
Inp
ut
Mo
de
l
Hierarchy Levels
![Page 24: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/24.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Tracking
![Page 25: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/25.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Tracking Comparison
![Page 26: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/26.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Tracking Comparison
![Page 27: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/27.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Tracking Comparison
![Page 28: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/28.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Reenactment
![Page 29: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/29.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
ReenactmentOnline RGB-Tracking
Preprocessed Video Tracking
Identity
Expression
Illumination
PoseP
er
Fram
e
Identity
Expression
Illumination
Pose
Pe
r Fr
ame
Reenactment
Expression Transfer
Mouth Retrieval
Compositing
Sou
rce
Act
or
Targ
et A
cto
r
![Page 30: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/30.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
ReenactmentOnline RGB-Tracking
Preprocessed Video Tracking
Identity
Expression
Illumination
PoseP
er
Fram
e
Identity
Expression
Illumination
Pose
Pe
r Fr
ame
Reenactment
Expression Transfer
Mouth Retrieval
Compositing
Sou
rce
Act
or
Targ
et A
cto
r
![Page 31: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/31.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Mouth-Retrieval
![Page 32: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/32.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Mouth-Retrieval
![Page 33: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/33.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Reenactment Comparison
![Page 34: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/34.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Live-Demo
![Page 35: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/35.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Limitations / Future Work
• Assumption of Lambertian surface and smooth illumination
• No occlusion handling
• No person specific details (fine scale details / wrinkles)
• Reenactment relies on a training sequence (Mouth retrieval)
![Page 36: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/36.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
Conclusion
• First Real-time Facial Reenactment only based on RGB-videos• Non-Rigid Model-Based Bundling
• Sub-Space Deformation Transfer
• Image-Based Mouth Synthesis
![Page 37: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/37.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
Thank You!
![Page 38: Face2Face: Real-time Face Capture and Reenactment of RGB … · 2016-08-08 · IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2Face: Real-time Face Capture and](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f03c10f7e708231d40a9aae/html5/thumbnails/38.jpg)
IEEE 2016 Conference on
Computer Vision and Pattern
Recognition
ResultsReenactmentFace CaptureFace Model
References• O. Alexander, M. Rogers, W. Lambeth, M. Chiang, and P. Debevec.
The Digital Emily Project: photoreal facial modeling and animation.In ACM SIGGRAPH Courses, pages 12:1–12:15. ACM, 2009.
• P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Perez, and C. Theobalt.Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track.In Computer Graphics Forum. Wiley-Blackwell, 2015.
• F. Shi, H.-T. Wu, X. Tong, and J. Chai.Automatic acquisition of high-fidelity facial performances using monocular videos.ACM TOG, 33(6):222, 2014.
• C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou.Facewarehouse: A 3D facial expression database for visual computing. IEEE TVCG, 20(3):413–425, 2014.
• J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt.Real-time expression transfer for facial reenactment.ACM Transactions on Graphics (TOG),34(6), 2015.
• V. Blanz and T. Vetter.A morphable model for the synthesis of 3d faces.In Proc. SIGGRAPH, pages 187–194. ACM Press/Addison-Wesley Publishing Co., 1999.