Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem,...
-
Upload
frederica-spencer -
Category
Documents
-
view
214 -
download
0
Transcript of Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem,...
![Page 1: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/1.jpg)
Image Stitching
Ali FarhadiCSEP 576
Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher
![Page 2: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/2.jpg)
• Combine two or more overlapping images to make one larger image
Add example
Slide credit: Vaibhav Vaish
![Page 3: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/3.jpg)
How to do it?
• Basic Procedure1. Take a sequence of images from the same
position1. Rotate the camera about its optical center
2. Compute transformation between second image and first
3. Shift the second image to overlap with the first4. Blend the two together to create a mosaic5. If there are more images, repeat
![Page 4: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/4.jpg)
1. Take a sequence of images from the same position
• Rotate the camera about its optical center
![Page 5: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/5.jpg)
2. Compute transformation between images
• Extract interest points• Find Matches• Compute transformation ?
![Page 6: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/6.jpg)
3. Shift the images to overlap
![Page 7: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/7.jpg)
4. Blend the two together to create a mosaic
![Page 8: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/8.jpg)
5. Repeat for all images
![Page 9: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/9.jpg)
How to do it?
• Basic Procedure1. Take a sequence of images from the same
position1. Rotate the camera about its optical center
2. Compute transformation between second image and first
3. Shift the second image to overlap with the first4. Blend the two together to create a mosaic5. If there are more images, repeat
✓
![Page 10: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/10.jpg)
Compute Transformations
• Extract interest points• Find good matches • Compute transformation
✓
Let’s assume we are given a set of good matching interest points
✓
![Page 11: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/11.jpg)
mosaic PP
Image reprojection
• The mosaic has a natural interpretation in 3D– The images are reprojected onto a common plane– The mosaic is formed on this plane
![Page 12: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/12.jpg)
Example
Camera Center
![Page 13: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/13.jpg)
Image reprojection
• Observation– Rather than thinking of this as a 3D reprojection, think
of it as a 2D image warp from one image to another
![Page 14: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/14.jpg)
Motion models
• What happens when we take two images with a camera and try to align them?
• translation?• rotation?• scale?• affine?• Perspective?
![Page 15: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/15.jpg)
Recall: Projective transformations
• (aka homographies)
![Page 16: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/16.jpg)
Parametric (global) warping
• Examples of parametric warps:
translation rotation aspect
affineperspective
![Page 17: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/17.jpg)
2D coordinate transformations
• translation: x’ = x + t x = (x,y)
• rotation: x’ = R x + t• similarity: x’ = s R x + t• affine: x’ = A x + t• perspective: x’ H x x = (x,y,1)
(x is a homogeneous coordinate)
![Page 18: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/18.jpg)
Image Warping
• Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?
f(x) g(x’)x x’
h(x)
![Page 19: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/19.jpg)
Forward Warping
• Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?
![Page 20: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/20.jpg)
Forward Warping
• Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?• Answer: add “contribution” to several pixels,
normalize later (splatting)
![Page 21: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/21.jpg)
Image Stitching 21Richard Szeliski
Inverse Warping
• Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
f(x) g(x’)x x’
h-1(x)
• What if pixel comes from “between” two pixels?
![Page 22: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/22.jpg)
Image Stitching 22Richard Szeliski
Inverse Warping
• Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)• What if pixel comes from “between” two pixels?• Answer: resample color value from interpolated source image
f(x) g(x’)x x’
h-1(x)
![Page 23: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/23.jpg)
Interpolation
• Possible interpolation filters:– nearest neighbor– bilinear– bicubic (interpolating)
![Page 24: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/24.jpg)
Motion models
Translation
2 unknowns
Affine
6 unknowns
Perspective
8 unknowns
3D rotation
3 unknowns
![Page 25: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/25.jpg)
Finding the transformation
• Translation = 2 degrees of freedom• Similarity = 4 degrees of freedom• Affine = 6 degrees of freedom• Homography = 8 degrees of freedom
• How many corresponding points do we need to solve?
![Page 26: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/26.jpg)
Plane perspective mosaics
– 8-parameter generalization of affine motion• works for pure rotation or planar surfaces
– Limitations:• local minima • slow convergence• difficult to control interactively
![Page 27: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/27.jpg)
Simple case: translations
How do we solve for ?
![Page 28: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/28.jpg)
Mean displacement =
Simple case: translations
Displacement of match i =
![Page 29: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/29.jpg)
Simple case: translations
• System of linear equations– What are the knowns? Unknowns?– How many unknowns? How many equations (per match)?
![Page 30: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/30.jpg)
Simple case: translations
• Problem: more equations than unknowns– “Overdetermined” system of equations– We will find the least squares solution
![Page 31: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/31.jpg)
Least squares formulation
• For each point
• we define the residuals as
![Page 32: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/32.jpg)
Least squares formulation
• Goal: minimize sum of squared residuals
• “Least squares” solution• For translations, is equal to mean displacement
![Page 33: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/33.jpg)
Least squares
• Find t that minimizes
• To solve, form the normal equations
![Page 34: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/34.jpg)
Solving for translations
• Using least squares
2n x 2 2 x 1 2n x 1
![Page 35: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/35.jpg)
Affine transformations
• How many unknowns?• How many equations per match?• How many matches do we need?
![Page 36: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/36.jpg)
Affine transformations
• Residuals:
• Cost function:
![Page 37: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/37.jpg)
Affine transformations
• Matrix form
2n x 6 6 x 1 2n x 1
![Page 38: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/38.jpg)
Solving for homographies
![Page 39: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/39.jpg)
Solving for homographies
![Page 40: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/40.jpg)
Direct Linear Transforms
Defines a least squares problem:
• Since is only defined up to scale, solve for unit vector• Solution: = eigenvector of with smallest eigenvalue• Works with 4 or more points
2n × 9 9 2n
![Page 41: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/41.jpg)
Image Stitching 41Richard Szeliski
Matching features
What do we do about the “bad” matches?
![Page 42: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/42.jpg)
Image Stitching 42Richard Szeliski
RAndom SAmple Consensus
Select one match, count inliers
![Page 43: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/43.jpg)
Image Stitching 43Richard Szeliski
RAndom SAmple Consensus
Select one match, count inliers
![Page 44: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/44.jpg)
Image Stitching 44Richard Szeliski
Least squares fit
Find “average” translation vector
![Page 45: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/45.jpg)
![Page 46: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/46.jpg)
Structure from Motion 46
RANSAC for estimating homography
• RANSAC loop:1. Select four feature pairs (at random)2. Compute homography H (exact)3. Compute inliers where ||pi’, H pi|| < ε
• Keep largest set of inliers• Re-compute least-squares H estimate using all
of the inliers
CSE 576, Spring 2008
![Page 47: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/47.jpg)
47
Simple example: fit a line
• Rather than homography H (8 numbers) fit y=ax+b (2 numbers a, b) to 2D pairs
47
![Page 48: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/48.jpg)
48
Simple example: fit a line
• Pick 2 points• Fit line• Count inliers
48
3 inliers
![Page 49: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/49.jpg)
49
Simple example: fit a line
• Pick 2 points• Fit line• Count inliers
49
4 inliers
![Page 50: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/50.jpg)
50
Simple example: fit a line
• Pick 2 points• Fit line• Count inliers
50
9 inliers
![Page 51: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/51.jpg)
51
Simple example: fit a line
• Pick 2 points• Fit line• Count inliers
51
8 inliers
![Page 52: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/52.jpg)
52
Simple example: fit a line
• Use biggest set of inliers• Do least-square fit
52
![Page 53: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/53.jpg)
RANSAC
Red: rejected by 2nd nearest neighbor criterion
Blue:Ransac outliers
Yellow:inliers
![Page 54: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/54.jpg)
How many rounds?
• If we have to choose s samples each time– with an outlier ratio e– and we want the right answer with probability p
![Page 55: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/55.jpg)
Image Stitching 55Richard Szeliski
Rotational mosaics
– Directly optimize rotation and focal length
– Advantages:• ability to build full-view
panoramas• easier to control interactively• more stable and accurate
estimates
![Page 56: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/56.jpg)
Image Stitching 56Richard Szeliski
Rotational mosaic
• Projection equations1. Project from image to 3D ray• (x0,y0,z0) = (u0-uc,v0-vc,f)
2. Rotate the ray by camera motion• (x1,y1,z1) = R01 (x0,y0,z0)
3. Project back into new (source) image• (u1,v1) = (fx1/z1+uc,fy1/z1+vc)
![Page 57: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/57.jpg)
Computing homography
• Assume we have four matched points: How do we compute homography H?
Normalized DLT1. Normalize coordinates for each image
a) Translate for zero meanb) Scale so that average distance to origin is ~sqrt(2)
– This makes problem better behaved numerically
2. Compute using DLT in normalized coordinates3. Unnormalize:
Txx ~ xTx ~
THTH~1
ii Hxx
H~
![Page 58: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/58.jpg)
Computing homography• Assume we have matched points with outliers: How do we
compute homography H?
Automatic Homography Estimation with RANSAC1. Choose number of samples N2. Choose 4 random potential matches3. Compute H using normalized DLT4. Project points from x to x’ for each potentially matching
pair:5. Count points with projected distance < t
– E.g., t = 3 pixels
6. Repeat steps 2-5 N times– Choose H with most inliers
HZ Tutorial ‘99
ii Hxx
![Page 59: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/59.jpg)
Automatic Image Stitching
1. Compute interest points on each image
2. Find candidate matches
3. Estimate homography H using matched points and RANSAC with normalized DLT
4. Project each image onto the same surface and blend
![Page 60: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/60.jpg)
RANSAC for Homography
Initial Matched Points
![Page 61: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/61.jpg)
RANSAC for Homography
Final Matched Points
![Page 62: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/62.jpg)
RANSAC for Homography
![Page 63: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/63.jpg)
Image Stitching 63Richard Szeliski
Determining the focal length
1. Initialize from homography H(see text or [SzSh’97])
2. Use camera’s EXIF tags (approx.)3. Use a tape measure
4. Ask your instructor
1m4m
![Page 64: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/64.jpg)
Assembling the panorama
• Stitch pairs together, blend, then crop
![Page 65: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/65.jpg)
Problem: Drift
• Error accumulation– small errors accumulate over time
![Page 66: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/66.jpg)
Problem: Drift
• Solution– add another copy of first image at the end– this gives a constraint: yn = y1
– there are a bunch of ways to solve this problem• add displacement of (y1 – yn)/(n -1) to each image after the first• compute a global warp: y’ = y + ax• run a big optimization problem, incorporating this constraint
– best solution, but more complicated– known as “bundle adjustment”
(x1,y1)
copy of first image
(xn,yn)
![Page 67: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/67.jpg)
Choosing a Projection Surface
Many to choose: planar, cylindrical, spherical, cubic, etc.
![Page 68: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/68.jpg)
Planar Mapping
f f
x
x
1) For red image: pixels are already on the planar surface2) For green image: map to first image plane
![Page 69: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/69.jpg)
Planar Projection
Planar
Photos by Russ Hewett
![Page 70: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/70.jpg)
Planar Projection
Planar
![Page 71: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/71.jpg)
Cylindrical Mapping
ff
xx
1) For red image: compute h, theta on cylindrical surface from (u, v)2) For green image: map to first image plane, than map to cylindrical surface
![Page 72: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/72.jpg)
Cylindrical ProjectionCylindrical
![Page 73: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/73.jpg)
Cylindrical ProjectionCylindrical
![Page 74: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/74.jpg)
Planar
Cylindrical
![Page 75: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/75.jpg)
Image Blending
![Page 76: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/76.jpg)
Feathering
01
01
+
=
![Page 77: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/77.jpg)
Effect of window (ramp-width) size
0
1 left
right0
1
![Page 78: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/78.jpg)
Effect of window size
0
1
0
1
![Page 79: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/79.jpg)
Good window size
0
1
“Optimal” window: smooth but not ghosted• Doesn’t always work...
![Page 80: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/80.jpg)
Pyramid blending
Create a Laplacian pyramid, blend each level• Burt, P. J. and Adelson, E. H., A multiresolution spline with applications to image mosaics, ACM Transactions on
Graphics, 42(4), October 1983, 217-236.
![Page 81: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/81.jpg)
Gaussian Pyramid Laplacian Pyramid
The Laplacian Pyramid
0G
1G
2GnG
- =
0L
- =1L
- = 2Lnn GL
)expand( 1 iii GGL
)expand( 1 iii GLG
![Page 82: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/82.jpg)
Laplacianlevel
4
Laplacianlevel
2
Laplacianlevel
0
left pyramid right pyramid blended pyramid
![Page 83: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/83.jpg)
Laplacian image blend
1. Compute Laplacian pyramid2. Compute Gaussian pyramid on weight image
(can put this in A channel)3. Blend Laplacians using Gaussian blurred
weights4. Reconstruct the final image• Q: How do we compute the original weights?
![Page 84: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/84.jpg)
Multiband Blending with Laplacian Pyramid
0
1
0
1
0
1
Left pyramid Right pyramidblend
• At low frequencies, blend slowly• At high frequencies, blend quickly
![Page 85: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/85.jpg)
Multiband blending
1. Compute Laplacian pyramid of images and mask
2. Create blended image at each level of pyramid
3. Reconstruct complete image
Laplacian pyramids
![Page 86: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/86.jpg)
Blending comparison (IJCV 2007)
![Page 87: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/87.jpg)
Poisson Image Editing
• For more info: Perez et al, SIGGRAPH 2003– http://research.microsoft.com/vision/cambridge/papers/perez_siggraph03.pdf
![Page 88: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/88.jpg)
Encoding blend weights: I(x,y) = (αR, αG, αB, α)
color at p =
Implement this in two steps:
1. accumulate: add up the (α premultiplied) RGBα values at each pixel
2. normalize: divide each pixel’s accumulated RGB by its α value
Alpha Blending
Optional: see Blinn (CGA, 1994) for details:http://ieeexplore.ieee.org/iel1/38/7531/00310740.pdf?isNumber=7531&prod=JNL&arnumber=310740&arSt=83&ared=87&arAuthor=Blinn%2C+J.F.
I1
I2
I3
p
![Page 89: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/89.jpg)
Choosing seams
Image 1
Image 2
x x
im1 im2
• Easy method– Assign each pixel to image with nearest center
![Page 90: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/90.jpg)
Choosing seams
• Easy method– Assign each pixel to image with nearest center– Create a mask: – Smooth boundaries ( “feathering”): – Composite
Image 1
Image 2
x x
im1 im2
![Page 91: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/91.jpg)
Choosing seams
• Better method: dynamic program to find seam along well-matched regions
Illustration: http://en.wikipedia.org/wiki/File:Rochester_NY.jpg
![Page 92: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/92.jpg)
Gain compensation• Simple gain adjustment
– Compute average RGB intensity of each image in overlapping region
– Normalize intensities by ratio of averages
![Page 93: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/93.jpg)
Blending Comparison
![Page 94: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/94.jpg)
De-Ghosting
![Page 95: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/95.jpg)
Recognizing Panoramas
Brown and Lowe 2003, 2007Some of following material from Brown and Lowe 2003 talk
![Page 96: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/96.jpg)
Recognizing Panoramas
Input: N images1. Extract SIFT points, descriptors from all
images2. Find K-nearest neighbors for each point (K=4)3. For each image
a) Select M candidate matching images by counting matched keypoints (m=6)
b) Solve homography Hij for each matched image
![Page 97: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/97.jpg)
Recognizing Panoramas
Input: N images1. Extract SIFT points, descriptors from all images2. Find K-nearest neighbors for each point (K=4)3. For each image
a) Select M candidate matching images by counting matched keypoints (m=6)
b) Solve homography Hij for each matched image
c) Decide if match is valid (ni > 8 + 0.3 nf )
# inliers # keypoints in overlapping area
![Page 98: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/98.jpg)
Recognizing Panoramas (cont.)
(now we have matched pairs of images)4. Find connected components
![Page 99: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/99.jpg)
Finding the panoramas
![Page 100: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/100.jpg)
Finding the panoramas
![Page 101: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/101.jpg)
Finding the panoramas
![Page 102: Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.](https://reader038.fdocuments.in/reader038/viewer/2022110400/56649da55503460f94a90b5d/html5/thumbnails/102.jpg)
Recognizing Panoramas (cont.)
(now we have matched pairs of images)4. Find connected components5. For each connected component
a) Solve for rotation and fb) Project to a surface (plane, cylinder, or sphere)c) Render with multiband blending