Alexandros Lattas Poster - UCL - London's Global University
19
AvatarMe: Realistically Renderable 3D Facial Reconstruction “in-the-wild” Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis, Vasileios Triantafyllou, Abhijeet Ghosh, Stefanos Zafeiriou
Transcript of Alexandros Lattas Poster - UCL - London's Global University
AvatarMe: Realistically Renderable 3D Facial Reconstruction “in-the-wild”
Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis,
Vasileios Triantafyllou, Abhijeet Ghosh, Stefanos Zafeiriou
Facial Reconstruction Head ReconstructionInput
Realistically Renderable 3D Facial Reconstruction “in-the-wild”AvatarMe
Realistically Renderable 3D Facial Reconstruction “in-the-wild”
Input
Cathedral + Point lights Sunset + Point light Underpass + Point lights
AvatarMe
Data
Light Stage Captures
Imperial College Multispectral Light Stage
Diffuse Albedo Diffuse Normals (Object Space)
Specular Albedo Specular Normals (Tangent Space)
Over 200 individuals captured using unpolarized binary1, 2 and polarized3 gradient illumination.
RealFaceDB: Visit github.com/lattas/AvatarMe
Data
Method
1. Base Reconstruction
Input
Reconstruction 𝑺
Completed Texture𝐓
Shape Normals (Object) 𝑵𝑶
Shape Normals (Tangent) 𝑵𝑻
Depth (Object) 𝑫𝑶
Method
3DMM fitting using GANFIT4:• GAN-generated texture• Deep identity features optimization
(576×384)
2. Inverse Rendering
Relighted UV map 𝑨𝑫𝑻 from captured reflectance
Estimated the point light sources (●) and camera (●) directions used for training and the environment
illumination using a Cornea model5.
Method
Reconstructed Sample 𝔼𝒕∈ 𝑻𝟏,𝑻𝟐,…,𝑻𝒏
Cornea Reflection
3. Super Resolution
Completed Texture 𝑻
(8×) Completed Texture 𝑻
Deep Residual Channel Attention Networks (RCAN)6, trained on GANFIT-like illuminated RealFaceDB data.
8×
Method
(768×512) (6144×4096)
𝜁(𝑻)
3. Reflectance InferenceMethod
Image translation network, based on
pix2pixHD6.
- Adversarial loss- Feature Matching Loss- No VGG perpetual loss
Texture 𝑻 Depth 𝑫𝑶 Diffuse Albedo 𝑨𝑫
[512, 512]
[Z][R, G, B]
Diffuse Albedo
(6144×4096)[R, G, B]
𝛿(𝑻,𝑫𝑶)
Delighting of the super-resolved textures 𝑻 from the previous step, trained on the relighted captured data 𝑨𝑫𝑻 .
4. Head Completion
Geometry 𝑺Diffuse Albedo 𝑨𝑫 Specular Albedo 𝑨𝑺 Specular Normals 𝑵𝑺
We regress the head geometry and translate textures to head topology using the Universal Head Model (UHM)7.
Method
OverviewMethod
[6144, 4096]
[6144, 4096]
Results
Rendered Head ReconstructionResults
Reconstruction DetailInput
Rendering in different Environments
Examples rendered with an environment and point light sources.
Cathedral + Point lights
Sunset + Point light
Results
Consistency with different inputs
Input Diffuse Albedo Specular Albedo Normals Rendering
Results
Ablation
Input Reconstruction Super Resolution Delighting Full Model
Results
Details of rendering after each step of AvatarMe:
github.com/lattas/AvatarMe
@alexlattas
Realistically Renderable 3D Facial Reconstruction “in-the-wild”AvatarMe
1. C. Kampouris, S. Zafeiriou, and A. Ghosh, “Diffuse-Specular Separation using Binary Spherical Gradient Illumination,” Eurographics Symposium on Rendering, 2018.
2. A. Lattas, M. Wang, S. Zafeiriou, and A. Ghosh, “Multi-view facial capture using binary spherical gradient illumination,” in ACM SIGGRAPH 2019 Posters, Los Angeles, California, Jul. 2019.
3. A. Ghosh, G. Fyffe, B. Tunwattanapong, J. Busch, X. Yu, and P. Debevec, “Multiview face capture using polarized spherical gradient illumination,” ACM Trans. Graph., vol. 30, no. 6, pp. Dec. 2011.
4. B. Gecer, S. Ploumpis, I. Kotsia, and S. Zafeiriou, “GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction,” Proceedings of the IEEE conference on computer vision and pattern recognition, , 2019.
5. K. Nishino and S. K. Nayar, “Eyes for relighting,” ACM Trans. Graph., vol. 23, no. 3, pp. 704–711, Aug. 2004.
6. T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Proceedings of the IEEE conference on computer vision and pattern recognition, Aug. 2018.
7. S. Ploumpis et al., “Towards a complete 3D morphable model of the human head,” arXiv:1911.08008 [cs], Feb. 2020.
8. A. Chen, Z. Chen, G. Zhang, Z. Zhang, K. Mitchell, and J. Yu, “Photo-Realistic Facial Details Synthesis from Single Image,” The IEEE International Conference on Computer Vision (ICCV), 2019.
9. S. Yamaguchi et al., “High-fidelity facial reflectance and geometry inference from an unconstrained image,” ACM Trans. Graph., vol. 37, no. 4, pp. 162:1–162:14, Jul. 2018.
References
