Style Transfer for Headshot Portraits - People | MIT...
Transcript of Style Transfer for Headshot Portraits - People | MIT...
![Page 1: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/1.jpg)
Style Transfer for Headshot PortraitsYiChang Shih Sylvain Paris Connelly Barnes
MIT CSAIL Adobe University of Virginia
William T. Freeman Frédo Durand
MIT CSAIL MIT CSAIL
![Page 2: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/2.jpg)
Professional portraits look better
Ordinary photo Professional photo
![Page 3: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/3.jpg)
The goal: make good portraits easy
• Make look like
• Transfer the style from the example photo • Automatic
Ordinary photo Professional photo
![Page 4: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/4.jpg)
We work on headshots
• What we match: retouching, texture, lighting
• What we do not match: pose, expression, clothing, focal length, aperture
![Page 5: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/5.jpg)
Preview our result
ExampleInput Output
![Page 6: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/6.jpg)
Hard problem: color transfer is not sufficient • Humans are intolerant to artifacts on faces
Input [HaCohen et al. 2010] (lighting and details
are missing)
Example Our method
![Page 7: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/7.jpg)
Related work: global transfer• Work well on landscapes
Input Model Output by Bae et al. [2006]
• Do not work as well on portraits
[Bae et al. 2006, Sunkavalli et al. 2010…]
![Page 8: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/8.jpg)
Related work: global transfer• Work well on landscapes
Input Model Output by Bae et al. [2006]
• Do not work as well on portraits
[Bae et al. 2006, Sunkavalli et al. 2010…]
![Page 9: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/9.jpg)
Related work: local style transfer• Time hallucination [Shih et al. 2013, Laffont et al. 2014]
Example imagesInput: afternoon Output: night
• Requires two images: before and after
![Page 10: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/10.jpg)
Related work: face enhancement
• Image restoration: deblurring, denoising …
ExamplesBlurred input face Output: deblurred face
• We focus on photographic stylization.
[Joshi et al. 2010, Shih et al. 2013 …]
![Page 11: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/11.jpg)
Problem statement• Input: a casual frontal portrait and an example
• Output: ‐ The input portrait rendered in the example style ‐ Automatic ‐ The style includes texture, tone, and color
![Page 12: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/12.jpg)
Key idea #1: local transfer
• Local: eyes, nose, skin, etc. are treated differently
Input Example
![Page 13: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/13.jpg)
Key idea #1: local transfer
• Local: eyes, nose, skin, etc. are treated differently
Input Example
![Page 14: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/14.jpg)
Key idea #2: multi‐scale transfer• Textures at different scales are treated differently
Portrait #1 Portrait #2
![Page 15: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/15.jpg)
Key idea #2: multi‐scale transfer• Textures at different scales are treated differently
Portrait #1 Portrait #2
![Page 16: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/16.jpg)
Overview of the algorithm1. Dense matching between the input and example2. Multiscale transfer of local statistics 3. Post processing on eyes and background
Input Example Step 1: matching Step 2: transfer Step 3: post processing
![Page 17: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/17.jpg)
Step 1: dense matching• Rigid warp + SIFT flow to align semantic features [Liu et al. 2008]
Input Example Warped example
![Page 18: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/18.jpg)
Step 2: multi‐scale local transfer
Input
Example
![Page 19: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/19.jpg)
Step 2: multi‐scale local transfer
Input
Example
1. Construct Laplacian stacks for the input and the example
![Page 20: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/20.jpg)
Step 2: multi‐scale local transfer
Input
Example
1. Construct Laplacian stacks for the input and the example
2. Local match at each scale
![Page 21: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/21.jpg)
Step 2: multiscale transfer of local statistics
Input
Example
1. Construct Laplacian stacks for the input and the example
3. Collapse the matched stacks to create the output of this step Output
2. Local match at each scale
![Page 22: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/22.jpg)
Step 2: multi‐scale local transfer
Input
Example
1. Construct Laplacian stacks for the input and the example
3. Collapse the matched stacks to create the output of this step Output
2. Local match at each scale
![Page 23: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/23.jpg)
Local energy
Example Laplacian Local energy ℓ
Gaussian kernel at this scale
![Page 24: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/24.jpg)
At each scale: match local energy
Example energyInput energy
![Page 25: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/25.jpg)
At each scale: match local energy
Compute the gain map
Example Laplacian
Input Laplacian
Local energy S[E]
Local energy S[I]
Gain map =
![Page 26: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/26.jpg)
At each scale: match local energy
Compute the gain map
Example Laplacian
Input Laplacian
Local energy S[E]
Local energy S[I]
Modulate the input Laplacian
Input Laplacian Gain map Output Laplacian
× =
Gain map =
![Page 27: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/27.jpg)
Robust transfer• Clamp the gain map to avoid artifacts caused by moles or glasses on the example
Input Example Our robust transferWithout robust transfer
![Page 28: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/28.jpg)
Laplacian using a face mask• Preserve the hair boundary using normalized convolution and a face mask
Input Example Without using the mask(the edges disappear)
Our method(the edges are preserved)
![Page 29: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/29.jpg)
Step 3: post‐processing• Adding eye highlights• Replacing the background
Input Example Without eye highlights Adding eye highlights(Our final result)
![Page 30: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/30.jpg)
Algorithm recap
Input Example Step 1. Dense alignment
![Page 31: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/31.jpg)
Algorithm recap
Input Example Step 1. Dense alignment
Step 2. Local transfer
![Page 32: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/32.jpg)
Algorithm recap
Input Example Step 1. Dense alignment
Step 2. Local transfer
Step 3.Eyes and
background
![Page 33: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/33.jpg)
Automatic example selection• Retrieve the best examples based on the face similarity between the input
Input The top three retrieved results
![Page 34: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/34.jpg)
Automatic example selection
Style transferred results using the top three examplesInput
• The results are robust to the example choices
![Page 35: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/35.jpg)
Results
Input Style 1 Style 2 Style 3
Examples are shown in the insets
![Page 36: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/36.jpg)
Close‐up
Input Example Output
![Page 37: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/37.jpg)
Example Output
![Page 38: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/38.jpg)
Input Style 1 Style 2 Style 3
More results
![Page 39: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/39.jpg)
Input Style 1 Style 2 Style 3
Outdoor input
![Page 40: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/40.jpg)
Extra results
Input Style 1 Style 2 Style 3
![Page 41: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/41.jpg)
Comparisons
Input Example Global transfer[Bae et al. 2006]
Our result
![Page 42: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/42.jpg)
Input Example Our method [Sunkavalli et al. 2010]
Histogram transfer [Reinhard et al. 2001] [Pitié et al. 2007] Photoshop Match Color
![Page 43: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/43.jpg)
Different success levels: good results• The inputs are well lit
Input Output
![Page 44: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/44.jpg)
Hard case• Matting (face mask) failure
Input Output
![Page 45: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/45.jpg)
Limitations
Input Failure output
• Require the input and the example to have similar facial attributes, e.g., skin color
• Cannot handle hard shadows on the input
Example
![Page 46: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/46.jpg)
Evaluation• 94 headshot inputs from Flickr
• Available on our website
![Page 47: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/47.jpg)
Extension to videos
![Page 48: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/48.jpg)
Conclusion• We introduce a style transfer algorithm tailored for headshot portraits.
• Based on multiscale transfer of local image statistics
Input Example Output
![Page 49: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/49.jpg)
Code and data are available
people.csail.mit.edu/yichangshih/portrait_web/
• Matlab code• Flickr evaluation dataset
![Page 50: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/50.jpg)
Acknowledgments• We thank Kelly Castro for discussing with us how he works and for his feedback, Michael Gharbi and Krzysztof Templin for being our portrait models.
• We acknowledge the funding from Quanta Computer and Adobe.
![Page 51: Style Transfer for Headshot Portraits - People | MIT CSAILpeople.csail.mit.edu/yichangshih/portrait_web/sig14... · 2014-08-29 · • Do not work as well on portraits [Bae et al.](https://reader034.fdocuments.in/reader034/viewer/2022050323/5f7ce204c98b2833ce5daa34/html5/thumbnails/51.jpg)
Conclusion• We introduce a style transfer algorithm tailored for headshot portraits.
• Based on multiscale transfer of local image statistics
Input Example Output