Object Removal in Multi-View Photos
description
Transcript of Object Removal in Multi-View Photos
![Page 1: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/1.jpg)
Object Removal in Multi-View Photos
Image Rectification
![Page 2: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/2.jpg)
Image Rectification
Figure 1: Example rectification of source images (1) to common image plane (2). 1
Transformation process used to project two-or-more images onto a common image plane.
Corrects image distortion by transforming the image into a standard coordinate system. 1
![Page 3: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/3.jpg)
Image Rectification
To perform a transform...
Cameras are calibrated and provide internal parameters resulting in an essential matrix representing relationship between the cameras.– We don’t have access to camera’s internal parameters.– What if single camera was used?
The more general case (without camera calibration) is represented by the fundamental matrix. 2
![Page 4: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/4.jpg)
Fundamental Matrix
Algebraic representation of epipolar geometry.
3×3 matrix which relates corresponding points in stereo images.
7 degrees of freedom, therefore at least 7 correspondences are required to compute the fundamental matrix. 3
![Page 5: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/5.jpg)
Corresponding Points
Figure out which parts of an image correspond to which parts of another image.– But what is a ‘part’ of an image?
‘part’ of an image is a Spatial Feature.
Spatial Feature Detection is the process of identifying spatial features in images.
![Page 6: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/6.jpg)
Spatial Feature Detection - Edges
Canny, Prewitt, Sobel, Difference of Gaussians...
Figure 2: Example application of Canny Edge Detection 4
![Page 7: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/7.jpg)
Spatial Feature Detection - Corners
Harris, FAST, SUSAN
Figure 2: Example application of Harris Corner Detection 5
![Page 8: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/8.jpg)
Feature Description
Simply identifying a feature point is not in itself useful.– consider how one would attempt to match detected
feature points between multiple images.
Scale-invariant feature transform (SIFT) offers robust feature description. 6
– Invariant to scale– Invariant to orientation– partially invariant to illumination changes
![Page 9: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/9.jpg)
SIFT
Uses Difference of Gaussians along with multiple smoothing and resampling filters to detect key points (Feature Points with descriptor data)
Key point specifies 2D location, scale, and orientation.
![Page 10: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/10.jpg)
SIFT
Figure 3: Sample image for SIFT application
![Page 11: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/11.jpg)
SIFT – Feature Points
Figure 4: Detected feature points via SIFT
![Page 12: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/12.jpg)
SIFT – Key Point
Figure 5: A SIFT key point in detail.
![Page 13: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/13.jpg)
SIFT - Matching
Matching SIFT key points by identifying nearest neighbour with the minimum Euclidean distance.
Ensures robustness via... Cluster identification by Hough transform voting. Model verification by linear least squares.
![Page 14: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/14.jpg)
SIFT - Matching
Figure 5: Example of matched SIFT key points. Note its tolerance to image scale and rotation.
![Page 15: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/15.jpg)
SIFT – Suitable for Multi-View?
SIFT fails to accurately match key points between images which vary significantly in perspective.
Figure 7 & 8: Comparison of SIFT accuracy with varying perspective angles.
Left image is 45 degrees with 152 matches.
Right image is 75 degrees with 11 matches.
![Page 16: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/16.jpg)
SIFT – Suitable for Multi-View?
SIFT fails to accurately match key points between images which undergo non-scalable affine transformation or projection.
Figure 9: SIFT fails to identify any key point matches between rotated images on a cylinder.
![Page 17: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/17.jpg)
ASIFT
A new framework for fully affine invariant image comparison.
Uses existing SIFT key point descriptors, but matching algorithm has improved.
![Page 18: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/18.jpg)
ASIFT – Improvements over SIFT
Simulated images are compared by a rotation, translation and zoom-invariant algorithm.– (SIFT normalizes translation and rotation and simulates
zoom.)
![Page 19: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/19.jpg)
ASIFT – Improvements over SIFT
Figure 10: ASIFT (left) identifies 165 matches compared to SIFT’s 11 on surface rotated 75 degrees
![Page 20: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/20.jpg)
ASIFT – Improvements over SIFT
Figure 10: ASIFT identifies 381 matches between rotated surfaces.
![Page 21: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/21.jpg)
Image Rectification
Quick Review...
1. Given multiple images of the same scene from different perspectives...
2. We have identified & matched feature points using ASIFT.
3. We now have the ability to calculate the fundamental matrix.
![Page 22: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/22.jpg)
Calculating Fundamental Matrix
![Page 23: Object Removal in Multi-View Photos](https://reader035.fdocuments.in/reader035/viewer/2022062520/56816108550346895dd05245/html5/thumbnails/23.jpg)
References
1. Oram, Daniel (2001). "Rectification for Any Epipolar Geometry“2. Fusiello, Andrea (2000-03-17). "Epipolar Rectification".
http://profs.sci.univr.it/~fusiello/rectif_cvol/rectif_cvol.html.3. Richard Hartley and Andrew Zisserman (2004). “Multiple View Geometry in
Computer Vision Second Edition”4. Ma,Yi. (1996) Basic Image Processing Demos (for EECS20)
http://robotics.eecs.berkeley.edu/~sastry/ee20/index.html 5. Mark Nixon & Alberto Aguado (2002), Feature Extraction & Image Processing,
Newnes6. Lowe, D. G., “Distinctive Image Features from Scale-Invariant Keypoints”,
International Journal of Computer Vision, 60, 2, pp. 91-110, 2004.