Multi-View Geometry (Ch7 New book. Ch 10/11 old...
Transcript of Multi-View Geometry (Ch7 New book. Ch 10/11 old...
![Page 1: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/1.jpg)
Multi-View Geometry(Ch7 New book.
Ch 10/11 old book)
Credits: M. Shah, UCF CAP5415, lecture 23 http://www.cs.ucf.edu/courses/cap6411/cap5415/, Trevor Darrell, Berkeley, C280, Marc Pollefeys
Guido GerigCS 6320 Spring 2015
![Page 2: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/2.jpg)
• Shading
Visual cues
![Page 3: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/3.jpg)
Visual cues
• Shading• Texture
The Visual Cliff, by William Vandivert, 1960
![Page 4: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/4.jpg)
Visual cues
From The Art of Photography, Canon
• Shading• Texture
• Focus
![Page 5: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/5.jpg)
Visual cues
• Shading• Texture
• Focus
• Motion
![Page 6: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/6.jpg)
Visual cues
• Shading• Texture
• Focus
• Motion• Shape From X
– X = (– shading, texture, – focus, motion, – rotation, ...)
Atmospheric Perspective
Linear Perspective
![Page 7: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/7.jpg)
Visual cues
Cornell CS569 S2008, Lecture 8, slide by Steve Marschnerhttp://www.cs.cornell.edu/courses/cs569/2008sp/about.stm
Shadows
![Page 8: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/8.jpg)
Visual cues
• Shading• Texture
• Focus
• Motion• Shape From X
– (X = shading, – texture, focus, – motion, rotation, ...)
• Stereo (disparity, multi-view)
![Page 9: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/9.jpg)
http://www.well.com/~jimg/stereo/stereo_list.htmlGrauman
![Page 10: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/10.jpg)
Stereo photography and stereo viewers
Invented by Sir Charles Wheatstone, 1838
Image courtesy of fisher‐price.com
Take two pictures of the same subject from two slightly different viewpoints and display so that each eye sees only one of the images.
Grauman
![Page 11: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/11.jpg)
http://www.johnsonshawmuseum.org
Grauman
![Page 12: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/12.jpg)
Human stereopsis: disparity
Disparity occurs when eyes fixate on one object; others appear at different visual angles
![Page 13: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/13.jpg)
Disparity: d = r‐l = D‐F.
Human stereopsis: disparity
Adapted from M. Pollefeys
![Page 14: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/14.jpg)
Disparity: d = r‐l = D‐F = 0.
Human stereopsis: disparity
Adapted from M. Pollefeys
F
D
r
l
![Page 15: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/15.jpg)
Example: Stereo to Depth Map
![Page 16: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/16.jpg)
J. M. Rehg © 2003
Stereo Vision
Left Right
baseline
Matching correlationwindows across scan lines
depth ),(),(
yxdBfyxZ
Z(x, y) is depth at pixel (x, y)d(x, y) is disparity
d(x, y)
![Page 17: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/17.jpg)
Pinhole Camera Model
ZXfx
![Page 18: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/18.jpg)
Basic Stereo Derivations
Derive expression for Z as a function of x1, x2, f and B
![Page 19: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/19.jpg)
Basic Stereo Derivations
ZXfx 1 Z
BfxZ
BXfx
12
21 xxfBZ
![Page 20: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/20.jpg)
Basic Stereo Derivations
Define the disparity: 21 xxd
dfBZ
![Page 21: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/21.jpg)
Standard stereo geometry
Disparityd:| |u’u
Baseline B
![Page 22: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/22.jpg)
Standard stereo geometry
Observations on disparity:• d shows large differences at small distances• d gets very small on large distances
![Page 23: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/23.jpg)
J. M. Rehg © 2003
Stereo Correspondence
• Search over disparity to find correspondences• Range of disparities to search over can change dramatically within a single image pair.
![Page 24: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/24.jpg)
Standard stereo geometry: Changes of ∆Z with ∆d
Observations:• at small d (far), ∆d corresponds to large ∆Z• at large d (close), ∆d corresponds to small ∆Z• important for analysis of precision/resolution
f’(d) = = -
![Page 25: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/25.jpg)
Standard stereo geometry: Changes of ∆d with ∆Z
Observations:• at small Z (close), ∆Z effects in large ∆d• at large Z (far), ∆Z effects in small ∆d• important for analysis of precision/resolution
df’(Z) = = -
![Page 26: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/26.jpg)
I1 I2 I10
Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.
Why is disparity important?
Given dense disparity map, we can calculate a depth/distance/range map.
![Page 27: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/27.jpg)
27
Goal: 3D from Stereo viaDisparity Map
F&PChapter 11
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 28: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/28.jpg)
Example: Stereo to Depth Map
![Page 29: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/29.jpg)
![Page 30: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/30.jpg)
Random dot stereograms
• Julesz 1960: Do we identify local brightness patterns before fusion (monocular process) or after (binocular)?
• To test: pair of synthetic images obtained by randomly spraying black dots on white objects
![Page 31: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/31.jpg)
Random dot stereograms
Forsyth & Ponce
![Page 32: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/32.jpg)
Random dot stereograms
![Page 33: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/33.jpg)
Random dot stereograms
![Page 34: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/34.jpg)
A Cooperative Model (Marr and Poggio, 1976)
Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
![Page 35: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/35.jpg)
Random dot stereograms
From Palmer, “Vision Science”, MIT Press
![Page 36: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/36.jpg)
Random dot stereograms• When viewed monocularly, they
appear random; when viewed stereoscopically, see 3d structure.
• Conclusion: human binocular fusion not directly associated with the physical retinas; must involve the central nervous system
• Imaginary* “cyclopean retina” that combines the left and right image stimuli as a single unit
Grauman
Visual Pathway.jpg wiki.ucl.ac.uk
*This was because it was as though we have a cyclopean eye inside our brains that can see cyclopean stimuli hidden to each of our actual eyes.
![Page 37: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/37.jpg)
Autostereograms
Images from magiceye.com
Exploit disparity as depth cue using single image
(Single image random dot stereogram, Single image stereogram)
![Page 38: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/38.jpg)
![Page 39: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/39.jpg)
Images from magiceye.com
Autostereograms
![Page 40: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/40.jpg)
Optical flowWhere do pixels move?
![Page 41: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/41.jpg)
Optical flowWhere do pixels move?
![Page 42: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/42.jpg)
http://www.johnsonshawmuseum.org
Grauman
![Page 43: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2015/Materials/CS6320-CV-F2015-chap11... · Goal: 3D from Stereo via Disparity Map F&P Chapter 11 image I(x,y) Disparity](https://reader034.fdocuments.in/reader034/viewer/2022051806/5ffc6cdf8afebe4135579cc8/html5/thumbnails/43.jpg)
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
Grauman