1Ellen L. Walker Stereo Vision Why? Two images provide information to extract (some) 3D information...
-
Upload
gideon-chivers -
Category
Documents
-
view
213 -
download
0
Transcript of 1Ellen L. Walker Stereo Vision Why? Two images provide information to extract (some) 3D information...
1 Ellen L. Walker
Stereo Vision
Why?
Two images provide information to extract (some) 3D information
We have good biological models (our own vision system)
Difficulties
Matching information from left to right … but we’ve already looked at some matching techniques … and some can take advantage of expectation
Calibrating the stereo rig Some methods require careful calibration Others avoid calibration entirely
2 Ellen L. Walker
Multiple Coordinate Frames
World Frame (Euclidean)
“Arbitrary” origin, z usually vertical
Camera Frame (Euclidean)
Focal point of the camera is the origin, Z points away from the image plane and is aligned with the optical axis.
Image Frame (Euclidean)
Axes X and Y aligned with camera frame. Origin is where the focal ray hits the image plane.
Image Frame (Affine)
Y, Z same as Camera frame, X maybe not perpendicular to Y (models non-rectangular pixels)
3 Ellen L. Walker
Perspective Projection Geometry (review)
Image plane
Focal point
f
y
Y
Z0
y/f = Y/Z y = fY/Z Z=fY/y
4 Ellen L. Walker
Triangulation
Given image point (x), and focal center (c), all possible world points lie along a ray with vector v:
Find the intersection of these rays to get the 3D point, (see section 7.1 for least-squares formulation)
Figure 7.1
€
ˆ v i = x i − c i( ) / x i − c i
5 Ellen L. Walker
Stereo Reconstruction
Epipolar geometry
Every point in one image, lies on a line in the other image
The epipolar line is the image of the ray from the focal point through the point
All epipolar lines pass through the epipole, which is the image of the focal point itself.
So what?
If cameras are calibrated, make epipolar lines line up on scan lines (epipole is at infinity) This is the “canonical configuration”)
If cameras are not calibrated, find the epipole and use it for calibration (8 point algorithm)
6 Ellen L. Walker
Epipolar Geometry
Figure 7.3
Epipolar points (e0 and e1), lines (l0 and l1) and corresponding points (x and x1)
7 Ellen L. Walker
Epipolar Geometry Definitions
c0, c1 - camera centers of focus
i0, i1 - image planes
p - point in space
x0, x1 – images of p
e0 – epipole 0 (image of c1 in i0) & vice versa for e1
Epipolar lines l0 and l1 connect e0 and x0; e1 and x1
Epipolar plane contains p, c0 and c1 (and all epipolar lines)
Epipolar constraint: All images of a point lie on its epipolar line.
8 Ellen L. Walker
Recovering the Epipolar Information
Begin by assuming 2 cameras are related by rotation R and translation T (We will not have to know R and T later)
Then:
(x0, y0, w0)T = P1 (X, Y, Z, 1)T Where P1 = (Id | 0) and Id is the 3x3 identity matrix
(x1, y1, w1)T = P2 (X, Y, Z, 1)T
Where P2 = (R | -RT)T and R and T are the rotation and translation matrices between the cameras
The image of the line that passes through 2 points (the camera origin C0 = (0, 0, 0, 1) and the point at infinity (x0, y0, z0, 0) is epipolar line 2
After some algebra (section 7.2), we get the important equation relating the points in the two images:
(x0, y0, w0) E (x1, y1, w1)T = 0E is the 3x3 essential matrix that relates the two images
9 Ellen L. Walker
Recovering the Epipolar Information
The equation (x0, y0, w0) Q (x1, y1, w1)T = 0 is true for every point that is visible in both images!
Since Q is a 3x3 matrix, we would need 9 linear equations to recover all 9 elements
But, we will never be able to recover absolute scale (since moving the camera closer is entirely equivalent to making the objects bigger)
Set Q[3][3] = 1
Use 8 correspondences to recover 8 points
"8 point algorithm" for epipolar constraint recovery
Given Q and p0 = (x0, y0, s0), the epipolar line is the set of all points P1 for which p0Qp1 = 0, which is an equation for the epipolar line!
10 Ellen L. Walker
Value of Epipolar Information
Recover epipole to use as a constraint for correspondence matching
Use epipolar information to warp images as they would appear in a calibrated rig (epipolar line -> x axis)
Recognize "possible / impossible" relationships among points based on epipolar constraints
Use the concept of "two views" in other ways
Object and shadow
Two copies of the same object (translational symmetry)
Surface of revolution (rotational symmetry of boundary curve)
11 Ellen L. Walker
Stereo in a Calibrated Rig
Assume cameras aligned on x axis, b and f known
Given xl and xr (and d = xl – xr), calculate Z
P = Xl,Z (left)
Cl
Cr
f
b
Xr = Xl – bZr = Zl = Z
xl = f Xl / Zxr = f Xr / Z = f(Xl – b)/Zxl – xr = (f/Z) (Xl – (Xl – b))xl – xr = f b / z
fxl
xr
12 Ellen L. Walker
Disparity Image
Given two rectified images (epipolar lines are horizontal or vertical), compute disparity (d) at each point
Disparity image (x, y ,d): x and y from image 0, d is the disparity
Distance is inversely proportional to disparity
Brighter points are closer
13 Ellen L. Walker
Finding Disparities
This is a matching problem
Use knowledge of camera setup to limit match locations
Along horizontal lines, for calibrated setup earlier
Along epipolar lines more generally
Matching strategies include:
Correlation (e.g. random dot stereogram)
(Point) feature extraction & matching
Object recognition & matching [not used by human vision]
Use of relational constraints (items don't trade places)
14 Ellen L. Walker
Sparse vs. Dense Stereo
Feature-based methods are sparse
First, find matchable features, then compute disparities via matches
Less computationally intensive (historically important)
Matches have high certainty
We want dense 3D information
Necessary for modeling, rendering
One way: use sparse matches as seeds, then fill in to make denser maps (analogs: region growing, thresholding with hysteresis)
15 Ellen L. Walker
Dense Stereo Taxonomy
Most methods perform the following steps:
Matching cost computation
Cost (support) aggregation
Disparity computation / optimization
Disparity refinement
“Cost” is generally with respect to an optimization framework (e.g. penalty for non-smoothness)
16 Ellen L. Walker
Sum of Squared Difference (local)
Matching cost is squared difference of intensity at given disparity (i.e. how different are the ‘matching’ pixels?)
Aggregation is adding up cost (at a given disparity) in a square window
Disparity selected based on minimum cost at each pixel
(Optional disparity refinement step can be added)
17 Ellen L. Walker
Optimization Algorithms (global)
Choose a local matching cost (similarity measure)
Apply global constraints (e.g. smoothness)
Use an optimization technique (e.g. simulated annealing, dynamic programming) to solve the resulting constrained optimization problem
Disparity refinement step can be added here
18 Ellen L. Walker
Dynamic Programming for Optimization
Row is left scanline, column is right scanline
Goal: generate least-cost diagonal path through matrix
M=match, L=left only, R=right only (L and R have fixed costs, M depends on match quality)
19 Ellen L. Walker
Disparity Refinement
For rendering, prevent ‘viewmaster’ appearance:
Objects seem to be aligned on fixed planes, e.g. cardboard cutouts stacked behind each other
Interpolate (“subpixel”) disparities to fit appropriate 3D curves and surfaces
Determine areas of occlusion (& verify)
Clean up noise with median filters, etc.
20 Ellen L. Walker
Segmentation Based Approach
First, segment the image into coherent regions
Oversegment to avoid mis-segmentation
Then, fit a local plane to each region
Iterative optimization technique, like relaxation
Allows for arbitrary discontinuities between regions
These techniques are best-ranked on Middlebury stereo evaluation site:
http://vision.middlebury.edu/stereo
21 Ellen L. Walker
Variations on Stereo
Trinocular stereo
Three calibrated cameras impose more constraints on correspondences
Multi-baseline stereo
When b is large, Z determination is more accurate "error diamonds" are not so elongated
When b is small, correspondences are easier to find
Sliding camera or 3 or more collinaer cameras allow both
(Depth estimate from small b constraints search in larger b)
22 Ellen L. Walker
Motion from 2D Image Sequences
Motion also gives multiple views
Multiple frames of translational motion similar to multiple-baseline images
Correspondence between sequential frames (small baseline)
Reconstruction using first and last frame (large baseline)
Camera moving on known path (e.g. into scene) allows reconstruction of unmoving objects from optical flow
Stable camera, single moving object Motion segmentation Trajectory estimation Possible 3D reconstruction depending on complexity of
object and trajectory
23 Ellen L. Walker
Stationary Object, Fixed Background
One or more discrete "moving objects" in the scene
Since most of the scene is stable, image subtraction will highlight objects
What changes are the leading & trailing edges
Changes are of opposite sign
Bounding box of moving object easy to determine
For best results, filter small noise regions
Smoothing before subtraction
Remove small regions of motion after subtraction
Closing to fill small gaps in moving objects' shapes
24 Ellen L. Walker
Optical Flow
Assume that intensity is not changing
Compute vector of each visible point between frames
Set of vectors is "optical flow field"
Issues
Computing point correspondences gives sparse field
Additional constraint from assuming consistent motion
Dense field computed as optimization with correlation and smoothness constraints
When object edges are not visible, only the motion normal to visible edges can be determined (aperture problem).
E.g. looking at a pole through a keyhole
25 Ellen L. Walker
Interpreting Optical Flow Field
Mostly 0, some regions of consistent vector
Translational object motion on stable background
Entire image is consistent vector
Translational camera motion in stable scene
Vectors pointing outward from a point
Motion into the scene towards that point, or expansion
Vectors pointing inward toward a point
Motion away from that point, or contraction
In all cases, larger vectors = faster motion
26 Ellen L. Walker
Range Sensing - Direct 3D
Structured light (visible, infrared, laser)
Simple case: replace second camera by a scanning laser - No correspondence problem!
More efficient: use stripes aligned with rows/columns; use patterns to avoid scanning
Active sensing (radar, sonar, laser, touch?)
Send out a signal & see how long it takes to bounce back
Use phase difference for more accurate data
Act on the object and record results (touch gives position and orientation of surface)