Image compositing and blending
Transcript of Image compositing and blending
![Page 1: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/1.jpg)
Computational Photography and Video:
Image compositing and blending
Prof. Marc Pollefeys
Dr. Gabriel Brostow
![Page 2: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/2.jpg)
Today’s schedule
• Last week’s recap
• Image pyramids
• Graphcuts
![Page 3: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/3.jpg)
Global parametric image warping
⎥⎥⎦
⎤
⎢⎢⎣
⎡
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
⎥⎥⎦
⎤
⎢⎢⎣
⎡
wyx
fedcba
wyx
100''' cos sin
' sin cos1 0 0 1 1
x
y
x t xy t y
Θ − Θ⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥= Θ Θ⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎥
⎥⎦
⎤
⎢⎢⎣
⎡
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
⎥⎥⎦
⎤
⎢⎢⎣
⎡
wyx
ihgfedcba
wyx
'''
·x> 0 −x0x>
0 x> −y0x>
¸⎡⎣ h>1h>2h>3
⎤⎦ = · 00
¸Compute homography: 2 equations/point
![Page 4: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/4.jpg)
Morphing = Object Averaging
• Morphing:– Identify corresponding points and determine triangulation– Linearly interpolate vertex coordinates and RGB image values
(vary coefficient continuously from 0 to 1)
![Page 5: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/5.jpg)
Schedule Computational Photography and Video
20 Feb Introduction to Computational Photography
27 Feb More on Cameras, Sensors and Color Assignment 1: Color
5 Mar Warping, morphing and mosaics Assignment 2: Alignment
12 Mar Image pyramids, Graphcuts Assignment 3: Blending
19 Mar Dynamic Range, HDR imaging, tone mapping Assignment 4: HDR
26 Mar Easter holiday – no classes
2 Apr TBD Project proposals
9 Apr TBD Papers
16 Apr TBD Papers
23 Apr TBD Papers
30 Apr TBD Project update
7 May TBD Papers
14 May TBD Papers
21 May TBD Papers
28 May TBD Final project presentation
![Page 6: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/6.jpg)
Image Compositing and Blending
Slides from Alexei Efros
© NASA
![Page 7: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/7.jpg)
Image Compositing
![Page 8: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/8.jpg)
Compositing Procedure1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)
Composite by David Dewey
2. Blend them into the composite (in the right order)
![Page 9: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/9.jpg)
Just replacing pixels rarely works
Problems: boundries & transparency (shadows)
Binary mask
![Page 10: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/10.jpg)
Two Problems:
Semi-transparent objects
Pixels too large
![Page 11: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/11.jpg)
Solution: alpha channel
• Add one more channel:– Image(R,G,B,alpha)
• Encodes transparency (or pixel coverage):– Alpha = 1: opaque object (complete coverage)
– Alpha = 0: transparent object (no coverage)
– 0<Alpha<1: semi‐transparent (partial coverage)
• Example: alpha = 0.3
Partial coverage or semi-transparency
![Page 12: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/12.jpg)
Alpha Blending
alphamask
Icomp = αIfg + (1-α)Ibg
shadow
![Page 13: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/13.jpg)
Multiple Alpha Blending
So far we assumed that one image (background) is opaque.
If blending semi‐transparent sprites (the “A over B” operation):
Icomp = αaIa + (1‐αa)αbIbαcomp = αa + (1‐αa)αb
Note: sometimes alpha is premultiplied:
im(αR,αG,αB,α):
Icomp = Ia + (1‐αa)Ib(same for alpha!)
![Page 14: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/14.jpg)
Alpha Hacking…
No physical interpretation, but it smoothes the seams
![Page 15: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/15.jpg)
Feathering
01
01
+
=Encoding as transparency
Iblend = αIleft + (1-α)Iright
α (1−α)
![Page 16: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/16.jpg)
Setting alpha: simple averaging
Alpha = .5 in overlap region
![Page 17: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/17.jpg)
Setting alpha: center seam
Alpha = logical(dtrans1>dtrans2)
![Page 18: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/18.jpg)
Setting alpha: blurred seam
Alpha = blurred
![Page 19: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/19.jpg)
Setting alpha: center weighting
Alpha = dtrans1 / (dtrans1+dtrans2)
Distancetransform
Ghost!
![Page 20: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/20.jpg)
Affect of Window Size
0
1 left
right0
1
![Page 21: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/21.jpg)
Affect of Window Size
0
1
0
1
![Page 22: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/22.jpg)
Good Window Size
0
1
“Optimal” Window: smooth but not ghosted
![Page 23: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/23.jpg)
What is the Optimal Window?
To avoid seamswindow = size of largest prominent feature
To avoid ghostingwindow <= 2*size of smallest prominent feature
Natural to cast this in the Fourier domain• largest frequency <= 2*size of smallest frequency• image frequency content should occupy one “octave” (power of two)
FFT
![Page 24: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/24.jpg)
What if the Frequency Spread is Wide
Idea (Burt and Adelson)– Compute Fleft = FFT(Ileft), Fright = FFT(Iright)
– Decompose Fourier image into octaves (bands)• Fleft = Fleft1 + Fleft2 + …
– Feather corresponding octaves Flefti with Frighti
• Can compute inverse FFT and feather in spatial domain
– Sum feathered octave images in frequency domain
• Better implemented in spatial domain
FFT
![Page 25: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/25.jpg)
Octaves in the Spatial Domain
Bandpass Images
Lowpass Images
![Page 26: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/26.jpg)
Pyramid Blending
0
1
0
1
0
1
Left pyramid Right pyramidblend
![Page 27: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/27.jpg)
Pyramid Blending
![Page 28: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/28.jpg)
laplacianlevel
4
laplacianlevel
2
laplacianlevel
0
left pyramid right pyramid blended pyramid
![Page 29: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/29.jpg)
Laplacian Pyramid: Blending
General Approach:1. Build Laplacian pyramids LA and LB from images A and B
2. Build a Gaussian pyramid GR from selected region R
3. Form a combined pyramid LS from LA and LB using nodes of GR as weights:• LS(i,j) = GR(I,j,)*LA(I,j) + (1‐GR(I,j))*LB(I,j)
4. Collapse the LS pyramid to get the final blended image
![Page 30: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/30.jpg)
Blending Regions
![Page 31: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/31.jpg)
Horror Photo
© david dmartin (Boston College)
![Page 32: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/32.jpg)
Results from CMU class (fall 2005)
© Chris Cameron
![Page 33: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/33.jpg)
Season Blending (St. Petersburg)
![Page 34: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/34.jpg)
Season Blending (St. Petersburg)
![Page 35: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/35.jpg)
Simplification: Two‐band Blending
• Brown & Lowe, 2003– Only use two bands: high freq. and low freq.
– Blends low freq. smoothly
– Blend high freq. with no smoothing: use binary alpha
![Page 36: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/36.jpg)
Low frequency (λ > 2 pixels)
High frequency (λ < 2 pixels)
2‐band Blending
![Page 37: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/37.jpg)
Linear Blending
![Page 38: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/38.jpg)
2‐band Blending
![Page 39: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/39.jpg)
Gradient Domain
• In Pyramid Blending, we decomposed our image into 2nd derivatives (Laplacian) and a low‐res image
• Let us now look at 1st derivatives (gradients):– No need for low‐res image
• captures everything (up to a constant)
– Idea: • Differentiate
• Blend
• Reintegrate
![Page 40: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/40.jpg)
Gradient Domain blending (1D)
Twosignals
Regularblending
Blendingderivatives
bright
dark
![Page 41: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/41.jpg)
Gradient Domain Blending (2D)
• Trickier in 2D:– Take partial derivatives dx and dy (the gradient field)
– Fidle around with them (smooth, blend, feather, etc)
– Reintegrate• But now integral(dx) might not equal integral(dy)
– Find the most agreeable solution• Equivalent to solving Poisson equation
• Can use FFT, deconvolution, multigrid solvers, etc.
![Page 42: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/42.jpg)
Perez et al., 2003
![Page 43: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/43.jpg)
Perez et al, 2003
Limitations:– Can’t do contrast reversal (gray on black ‐> gray on white)
– Colored backgrounds “bleed through”
– Images need to be very well aligned
editing
![Page 44: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/44.jpg)
Don’t blend, CUT!
So far we only tried to blend between two images. What about finding an optimal seam?
Moving objects become ghosts
![Page 45: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/45.jpg)
Davis, 1998
Segment the mosaic– Single source image per segment
– Avoid artifacts along boundriesDijkstra’s algorithm
![Page 46: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/46.jpg)
min. error boundary
Minimal error boundaryoverlapping blocks vertical boundary
_ =2
overlap error
![Page 47: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/47.jpg)
Graphcuts
What if we want similar “cut‐where‐things‐agree” idea, but for closed regions?– Dynamic programming can’t handle loops
![Page 48: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/48.jpg)
Graph cuts (simple example à la Boykov&Jolly, ICCV’01)
n-links
s
t a cuthard constraint
hard constraint
Minimum cost cut can be computed in polynomial time
(max-flow/min-cut algorithms)
![Page 49: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/49.jpg)
Kwatra et al, 2003
Actually, for this example, DP will work just as well…
![Page 50: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/50.jpg)
Lazy Snapping
Interactive segmentation using graphcuts
![Page 51: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/51.jpg)
Putting it all together
Compositing images/mosaics– Have a clever blending function
• Feathering
• Center‐weighted
• blend different frequencies differently
• Gradient based blending
– Choose the right pixels from each image• Dynamic programming – optimal seams
• Graph‐cuts
Now, let’s put it all together:– Interactive Digital Photomontage, 2004 (video)
![Page 52: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/52.jpg)
![Page 53: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/53.jpg)
Image resizing
• Dynamic Range, HDR imaging, tone mapping
![Page 54: Image compositing and blending](https://reader033.fdocuments.in/reader033/viewer/2022050901/589045e31a28abf3618bc414/html5/thumbnails/54.jpg)
Next week
• Dynamic Range, HDR imaging, tone mapping
Before After