Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical...
Transcript of Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical...
![Page 1: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/1.jpg)
Matting and Compositing
Digital Visual Effectsg ffYung-Yu Chuang
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
Photomontage Photomontage
The Two Ways of Life, 1857, Oscar Gustav Rejlander y jPrinted from the original 32 wet collodion negatives.
![Page 2: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/2.jpg)
Photographic compositionsPhotographic compositions
Lang Ching-shan
Use of mattes for compositingUse of mattes for compositing
The Great Train Robbery (1903) matte shot
Use of mattes for compositingUse of mattes for compositing
The Great Train Robbery (1903) matte shot
Optical compositingOptical compositing
King Kong (1933) Stop-motion + optical compositing
![Page 3: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/3.jpg)
Digital matting and compositingDigital matting and compositing
The lost world (1925) The lost world (1997)
Miniature, stop-motion Computer-generated images
Digital matting and compostingDigital matting and composting
King Kong (1933) Jurassic Park III (2001)
Optical compositingBlue-screen matting, digital composition, Optical compositing digital composition,
digital matte painting
Smith Duff Catmull Porter
O d 1996Oscar award, 1996
Titanic
M tti d C itiMatting and Compositing
![Page 4: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/4.jpg)
backgroundreplacement
background
M tti d C iti
editing
Matting and Compositing
Digital matting: bluescreen mattingDigital matting: bluescreen matting
Forrest Gump (1994)
• The most common approach for films.• Expensive, studio setup.p p• Not a simple one-step process.
Color difference method (Ultimatte)Color difference method (Ultimatte)C=F+B F
Blue-screenphotograph
Spill suppressionif B>G then B=G
Matte creation=B-max(G,R)p g p (G, )
demo with Paint Shop Pro (B=min(B,G))
Problems with color differenceProblems with color difference
Background color is usually not perfect! (lighting, shadowing…)
![Page 5: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/5.jpg)
Chroma-keying (Primatte)Chroma keying (Primatte) Chroma-keying (Primatte)Chroma keying (Primatte)
demo
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
F BF B
foreground color alpha matte background plate
BFC )α(α 1CcompositeF BFC )α(α 1
compositingC
C iti
p gequationB =0
Compositing
![Page 6: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/6.jpg)
F BF B
BFC )α(α 1CcompositeF BFC )α(α 1
compositingC
C iti
p gequationB
C=1
Compositing
F BF B
BFC )α(α 1CcompositeFC
BFC )α(α 1
compositing
C iti
p gequationB =0.6
Compositing
F BF B
CobservationBFC )α(α 1 BFC )α(α 1
compositing
M tti
p gequation
Matting
F BF B
CBFC )α(α 1
Three approaches:
1 d # k BFC )α(α 1
compositing
1 reduce #unknowns
2 add observations
M tti
p gequation3 add priors
Matting
![Page 7: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/7.jpg)
F BBF BB
CBFC )α(α 1 BFC )α(α 1
difference
M tti ( d # k )
differencematting
Matting (reduce #unknowns)
F F
CBFC )α(α 1
BBFC )α(α 1
blue screen
M tti ( d # k )
blue screenmatting
Matting (reduce #unknowns)
F F
BFC )α(α 1BC
BFC )α(α 1BFC )α(α 1
M tti ( dd b ti )triangulation
Matting (add observations)
F BF B
BCBFC )α(α 1BG BFC )α(α 1
FGunknown
Natural image mattingM tti ( dd i )rotoscopingRuzon-Tomasi
unknown
Natural image mattingMatting (add priors)
![Page 8: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/8.jpg)
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
para observedf(z)+z ypara-
metersobserved
signal
)|(max* yzPz Example: super-resolution)|(max yzPz
z)()|(max zPzyP
super-resolutionde-blurringde-blocking
)(max
yPz de-blocking
…
B i f k
)()|(max zLzyLz
Bayesian framework
para observedf(z)+z ypara-
metersobserved
signal
)()|(max* zLzyLz )()|(max zLzyLzz
2)(zfy data a-priori
B i f k
2evidence knowledge
Bayesian framework
posterior probability
likelihood priors
B i f kBayesian framework
![Page 9: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/9.jpg)
P iPriors B i ttiBayesian matting
repeat1. fix alpha
2. fix F and B
O ti i tiuntil converge
Optimization
![Page 10: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/10.jpg)
![Page 11: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/11.jpg)
![Page 12: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/12.jpg)
![Page 13: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/13.jpg)
![Page 14: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/14.jpg)
DDemo
inputtrimapalphainputtrimapalpha
R ltResults
input compositeinput composite
R ltResults
input imagetrimap
C iComparisons
![Page 15: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/15.jpg)
Bayesian Ruzon-Tomasi
C iComparisons
Bayesian Ruzon-Tomasi
C iComparisons
MishimaMishima
C iComparisons
BayesianBayesian
C iComparisons
![Page 16: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/16.jpg)
input image
C iComparisons
Bayesian Mishima
C iComparisons
Bayesian Mishima
C iComparisons
i tinputvideo
Vid ttiVideo matting
![Page 17: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/17.jpg)
i tinputvideo
inputkeytrimaps
Vid ttiVideo matting
i tinputvideo
interpo-latedtrimaps
Vid ttiVideo matting
inp tinputvideo
interpo-t tlated
trimapsoutputalpha
Vid ttiVideo matting
Compo-inp t Compo-site
inputvideo
interpo-t tlated
trimapsoutputalpha
Vid ttiVideo matting
![Page 18: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/18.jpg)
optical flow
optical flow
![Page 19: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/19.jpg)
![Page 20: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/20.jpg)
S l itSample composite
![Page 21: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/21.jpg)
G b ttGarbage mattes G b ttGarbage mattes
B k d ti tiBackground estimation B k d ti tiBackground estimation
![Page 22: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/22.jpg)
Al h ttAlpha matte
without b k d
withb k d
C i
background background
Comparison
![Page 23: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/23.jpg)
![Page 24: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/24.jpg)
C
P(F)
B
![Page 25: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/25.jpg)
Problems with Bayesian mattingProblems with Bayesian matting
• It requires fine trimaps for good results• It requires fine trimaps for good results• It is tedious to generate fine trimaps• Its performance rapidly degrades when
foreground and background patterns foreground and background patterns become complex Th i di t d l l t l t th • There is no direct and local control to the resulting mattes
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
Scribble based inputScribble-based input
trimap scribblep
MotivationMotivation
![Page 26: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/26.jpg)
LazySnappingLazySnapping
L S in-th mean foreground color
LazySnapping
L S iLazySnapping L S iLazySnapping
![Page 27: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/27.jpg)
Matting approachesMatting approaches• Sampling approaches: solve for each • Sampling approaches: solve for each
alpha separately by utilizing local fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian mattingKnockout and Bayesian matting.
• Propagation approaches: solve the whole matte together by optimizing, e g Poisson BP random walker e.g. Poisson, BP, random walker, closed-form and robust matting.
Poisson mattingPoisson matting
Poisson mattingPoisson matting Robust mattingRobust matting• Jue Wang and Michael Cohen CVPR • Jue Wang and Michael Cohen, CVPR
2007
![Page 28: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/28.jpg)
Robust mattingRobust matting• Instead of fitting models a non-• Instead of fitting models, a non-
parametric approach is used
B i R bBayesian Robust
Robust mattingRobust matting• We must evaluate hypothesized • We must evaluate hypothesized
foreground/background pairs
Bj C
Fidistance ratio
F
Robust mattingRobust matting• To encourage pure fg/bg pixels add • To encourage pure fg/bg pixels, add
weights
B F1C
FF2
Robust mattingRobust matting• Combine them together Pick up the • Combine them together. Pick up the
best 3 pairs and average themconfidence
![Page 29: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/29.jpg)
Robust mattingRobust matting Robust mattingRobust matting
matte confidencematte confidence
Matte optimizationMatte optimization
Solved by Random Walk Algorithm
Matte optimizationMatte optimizationdata constraintsdata constraints
neighborhood constraints
![Page 30: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/30.jpg)
Demo (EZ Mask)Demo (EZ Mask) EvaluationEvaluation• 8 images collected in 3 different • 8 images collected in 3 different
ways• Each has a “ground truth” matte
EvaluationEvaluation• Mean square error is used as the • Mean square error is used as the
accuracy metric• Try 8 trimaps with different accuracy
for testing robustness for testing robustness • 7 methods are tested: Bayesian, y ,
Belief propagation, Poisson, Random Walk KnockOut2 Closed Form and Walk, KnockOut2, Closed-Form and Robust mattingg
![Page 31: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/31.jpg)
Quantitative evaluationQuantitative evaluation Subjective evaluationSubjective evaluation
Subjective evaluationSubjective evaluation Ranks of these algorithmsRanks of these algorithms
Poissonaccuracy
6 9robustness
6 8PoissonRandom walk
6.96.0
6.84.4
Knockout2Bayesian
4.53 9
4.56 0Bayesian
Belief PropagationClose form
3.93.32 6
6.03.12 0Close-form
Robust matting2.61.0
2.01.3
![Page 32: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/32.jpg)
SummarySummary• Propagation-based methods are more • Propagation-based methods are more
robust• Sampling-based methods often
generate more accurate mattes than generate more accurate mattes than propagation-based ones with fine trimaps Robust matting combines strengths • Robust matting combines strengths of both
New evaluation (CVPR 2009)New evaluation (CVPR 2009)• http://www alphamatting com/• http://www.alphamatting.com/
Soft scissorSoft scissor• Jue Wang et al SIGGRAPH 2007• Jue Wang et. al., SIGGRAPH 2007• Users interact in a similar way to y
intelligent scissors
FlowchartFlowchart
![Page 33: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/33.jpg)
FlowchartFlowchart Soft scissorSoft scissor
Demo (Power Mask)Demo (Power Mask) OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
![Page 34: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/34.jpg)
Matting with multiple observationsMatting with multiple observations• Invisible lightsg
– Polarized lightsInfrared– Infrared
• Thermo-key• Depth Keying (ZCam)• Flash matting• Flash matting
I i ibl li ht (I f d)Invisible lights (Infared)
I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (I f d)Invisible lights (Infared)
![Page 35: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/35.jpg)
I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (I f d)Invisible lights (Infared)
I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (P l i d)Invisible lights (Polarized)
![Page 36: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/36.jpg)
I i ibl li ht (P l i d)Invisible lights (Polarized) Th KThermo-Key
Th KThermo-Key ZCZCam
![Page 37: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/37.jpg)
D f ttiDefocus mattingvideo
M tti ith video
Matting with camera arrays
flash no flash matteflash no flash matte
Fl h ttiFlash matting
![Page 38: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/38.jpg)
Background is much further than foreground and receives almost no flash light
Fl h ttiFlash matting
Foreground flash matting equationg g q
Generate a trimap and directly apply Bayesian matting.
Fl h ttiFlash matting
F d fl h ttiForeground flash matting J i t B i fl h ttiJoint Bayesian flash matting
![Page 39: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/39.jpg)
J i t B i fl h ttiJoint Bayesian flash matting
flash no flash
C iComparison
foreground flash matting
ioint Bayesian flash matting
C iComparison Fl h ttiFlash matting
![Page 40: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/40.jpg)
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
ConclusionsConclusions• Matting algorithms improves a lot in • Matting algorithms improves a lot in
these 10 years d ll l • In production, it is still always
preferable to shoot against uniform p gbackgroundsAlgorithms for more complex • Algorithms for more complex backgrounds
• Devices or algorithms for automatic mattingmatting
Thanks for your attention!Thanks for your attention! Shadow mattingand composting
![Page 41: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/41.jpg)
target backgroundsource scene target backgroundblue screen image
target backgroundblue screen composite photographblue screen composite
![Page 42: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/42.jpg)
photographblue screen composite
G t i Geometric errors
photographblue screen composite
Ph t t i Photometric errors
![Page 43: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/43.jpg)
S LS L
C
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d iti tiShadow compositing equation
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d ttiShadow matting
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d ttiShadow matting
![Page 44: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/44.jpg)
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d ttiShadow matting
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d itiShadow compositing
![Page 45: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/45.jpg)
![Page 46: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/46.jpg)
G t i Geometric errors
target backgroundsource scene
target backgroundsource scene
R i t #1Requirement #1
target backgroundsource scene
R i t #2Requirement #2
![Page 47: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/47.jpg)
![Page 48: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/48.jpg)
![Page 49: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/49.jpg)
![Page 50: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/50.jpg)
![Page 51: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/51.jpg)
![Page 52: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/52.jpg)
![Page 53: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/53.jpg)
![Page 54: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/54.jpg)
![Page 55: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/55.jpg)
![Page 56: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/56.jpg)
E i t ttiEnvironment matting
photographblue screen matting
traditional compositing equation
BFC T
BFC T
backgroundforegroundcomposite
g
environment compositing equation [Zongker’99]
RA
BFC T
BFC T
backgroundforegroundcomposite
g
![Page 57: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/57.jpg)
O(k) images
E i t tti [Z k ’99]Environment matting [Zongker’99]
photographZongker et al.
P bl l di iProblem: color dispersion
photographZongker et al.
P bl l fProblem: glossy surface
photographZongker et al.
P bl lti l iProblem: multiple mappings
![Page 58: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/58.jpg)
WW
C TC Tweighting backgroundweightingfunction
gcomposite
A bit i hti f tiM lti d l i t d G iArbitrary weighting functionMultimodal oriented Gaussian
![Page 59: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/59.jpg)
high accuracy photographZongker et al.high accuracyalgorithm
P bl l di iProblem: color dispersion
high accuracy photographZongker et al.high accuracyalgorithm
Gl fGlossy surface
withoutwithoutwith photographwithoutorientation
withoutorientation
withorientation
withoutorientationorientation
O i t d G iOriented Gaussian
high accuracy photographZongker et al.high accuracyalgorithm
P bl lti l iProblem: multiple mappings
![Page 60: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/60.jpg)
WR
A
BFC T
BFC T
backgroundweightingforegroundcomposite
bac g ou dweightingfunction
3 3 1 433
3 observations
3 1 43
3 observations11 variables
A RA, R F
433 433
3 observations7 variables3 observations
A RA, R F
![Page 61: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/61.jpg)
413 413
3 observations5 variables3 observations
A R A A, R
A, colorless
F
T(c c )13T(cx , cy)
1113
3 observations
11
3 variables3 observations
A R A cx cy A, R
A, colorless
cx , cy , specularly
F refractive
Stimulus functionStimulus functionx
Stimulus functionStimulus functionx
yA
y
T( )M(T , A) T(cx, cy)
(cx, cy)
T
Ideal plane in RGB cubeIdeal plane in RGB cubeIdeal plane in RGB cubeIdeal plane in RGB cube
x
y
T(x y)T(x,y)
![Page 62: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/62.jpg)
Calibrated manifold in RGB cubeCalibrated manifold in RGB cubeCalibrated manifold in RGB cubeCalibrated manifold in RGB cube
x (cx ,cy)C (cx ,cy)C
T(x y)y
T(x,y)
Estimate c c and Estimate c c and
W
Estimate cx ,cy and Estimate cx ,cy and
backgroundif ld
W
Tmanifold
C (cx ,cy)
C = KC
C
KC
KK
Problem: noisy matteProblem: noisy matte Edge-preserving filteringEdge-preserving filteringEdge preserving filteringEdge preserving filtering
with filteringwithout filtering gg
![Page 63: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/63.jpg)
Input imageInput image Difference thresholdingDifference thresholding
Morphological operationMorphological operation FeatheringFeathering
![Page 64: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/64.jpg)
Heuristics for specular highlightsHeuristics for specular highlights
Wbackgroundmanifold
W
Tmanifold
CC >1+ C >1+
K
Heuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlights
cx cy
C=T(c c )C=T(cx, cy)
Heuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlights
==
input estimation foregroundinput estimation foreground(highlights)
![Page 65: Matting and Compositingcyy/courses/vfx/14spring/...The Great Train Robbery (1903) matte shot Optical compositing King Kong (1933) Stop-motion + optical compositing Digital matting](https://reader034.fdocuments.in/reader034/viewer/2022051604/5ffaa418ffea58772a54cbc9/html5/thumbnails/65.jpg)
Composite with highlightsComposite with highlights
mattingmethod
compositingmodel methodmodel
color blue-screenBFC )α(α 1
color blending
blue-screenBayesian
shadowShadowmattingβ)L(βSC 1
refraction
matting
High accuracyrefractionreflection
High-accuracyenv. matting WBFC