Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method...
Transcript of Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method...
![Page 1: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/1.jpg)
Stereo: Metrics, RectificationPlanesweep Stereo
Scott WehrweinCSCI 497P/597P: Computer Vision
![Page 2: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/2.jpg)
Announcements
• Reminder: Exam – out this morning– due Tuesday night
• P1– code due tonight– artifact due Tuesday night
![Page 3: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/3.jpg)
Goals
• Understand the basic metrics used to compare patches (SSD, SAD, NCC)
• Understand how to rectify a pair of stereo images given their intrinsics and extrinsics.
• Understand the plane sweep stereo algorithm.
![Page 4: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/4.jpg)
A Stereo Algorithm
1. For every pixel (x, y)1. For every disparity d
1. Get patch from image 1 at (x, y)2. Get patch from image 2 at (x + d, y)3. Compute cost using your metric of choice
C = np.array(h,w,D)for r in range(0,h):for c in range(0,w):for d in range(min_d, max_d):C[r,c,d] = metric(get_patch(im1,r,c), get_patch(im2,r,c+d)
disp = np.argmin(C, axis=2)depth = f * b / disp
![Page 5: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/5.jpg)
Metrics for Stereo Matching
• SSD = sum of squared differences
• SAD = sum of absolute differences
• NCC = normalized cross-correlation– (more convolutioncross correlation!)
![Page 6: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/6.jpg)
Un-Normalized Cross CorrelationInsight: a cross-correlation filter is good at finding patches that look like itself.
![Page 7: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/7.jpg)
Normalized Cross Correlation
subtract the mean of each vector:
Approach: apply a patch from one image as a filter across the other.Trick: normalize patches before computing product to add invariance.
![Page 8: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/8.jpg)
Stereo matching based on SSD
SSD
dmin dBest matching disparity
![Page 9: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/9.jpg)
left image band
right image band
cross correlation
1
0
0.5
x
Stereo with NCC:The Good Case
![Page 10: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/10.jpg)
left image band
right image band
cross correlation
1
0
x
0.5
target region
Stereo with NCC:The Bad Case
![Page 11: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/11.jpg)
Stereo results– Data from University of Tsukuba– Similar results on other images without ground truth
Ground truthScene
![Page 12: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/12.jpg)
Results with window search
Window-based matching(best window size)
Ground truth
![Page 13: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/13.jpg)
Better methods exist...
Fancier methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
For the latest and greatest: http://www.middlebury.edu/stereo/
![Page 14: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/14.jpg)
Rectified Stereo Cameras
![Page 15: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/15.jpg)
What if the cameras aren’t rectified?
If cameras are calibrated, i.e., we know:
…then we can rectify the stereo pair…or we can use plane sweep stereo
![Page 16: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/16.jpg)
Rectifying cameras
![Page 17: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/17.jpg)
Rectifying cameras
![Page 18: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/18.jpg)
Rectifying cameras
![Page 19: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/19.jpg)
Rectifying cameras
![Page 20: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/20.jpg)
Rectifying cameras
![Page 21: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/21.jpg)
Example: A rectified stereo pair
![Page 22: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/22.jpg)
GOTO notes
![Page 23: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/23.jpg)
Plane Sweep Stereo
![Page 24: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/24.jpg)
A Stereo Algorithm
1. For every pixel (x, y)1. For every disparity d
1. Get patch from image 1 at (x, y)2. Get patch from image 2 at (x + d, y)3. Compute cost using your metric of choice
C = np.array(h,w,d)for r in range(0,h):for c in range(0,w):for d in range(-maxd, maxd):C[r,c,d] = metric(get_patch(im1,r,c), get_patch(im2,r,c+d)
disp = np.max(C, axis=2)depth = f * b / disp
![Page 25: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/25.jpg)
C = np.array(h,w,d)for d in range(-maxd, maxd):for r in range(0,h):for c in range(0,w):C[r,c,d] = metric(get_patch(im1,r,c), get_patch(im2,r,c+d)
disp = np.max(C, axis=2)depth = f * b / disp
Plane Sweep Stereo Algorithm
1. For every disparity d1. For every pixel (x, y)
1. Get patch from image 1 at (x, y)2. Get patch from image 2 at (x + d, y)3. Compute cost using your metric of choice
![Page 26: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/26.jpg)
![Page 27: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/27.jpg)
![Page 28: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/28.jpg)
![Page 29: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/29.jpg)
![Page 30: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/30.jpg)
![Page 31: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/31.jpg)
Plane Sweep Stereo
![Page 32: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/32.jpg)
Active stereo with structured light
• Project “structured” light patterns onto the object– simplifies the correspondence problem– basis for active depth sensors, such as Kinect and iPhone X (using IR)
camera 2
camera 1
projector
camera 1
projector
Li Zhang’s one-shot stereo
![Page 33: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/33.jpg)
Other methods for getting depth
![Page 34: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/34.jpg)
Laser scanning
• Optical triangulation– Project a single stripe of laser light– Scan it across the surface of the object– This is a very precise version of structured light scanning
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
![Page 35: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/35.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
![Page 36: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/36.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
![Page 37: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/37.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
![Page 38: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/38.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
![Page 39: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/39.jpg)
Microsoft Kinect
![Page 40: Scott Wehrwein - facultyweb.cs.wwu.eduwehrwes/courses/csci497p_20f/lectures/L22.pdfFancier method Boykovet al., Fast Approximate Energy Minimization via Graph Cuts, International Conference](https://reader035.fdocuments.in/reader035/viewer/2022081620/6119368b7a9a0261b0253482/html5/thumbnails/40.jpg)
Active stereo with structured light
https://ios.gadgethacks.com/news/watch-iphone-xs-30k-ir-dots-scan-your-face-0180944/