Structure from images

Calibration

Review: Pinhole Camera

Review: Perspective Projection

Review: Perspective Projection

Points go to Points Lines go to Lines Planes go to whole

image or Half-planes

Polygons go to Polygons

Review: Intrinsic Camera Parameters

X

Y

Z C

Image plane

Focal plane

M

m

CCC ZYX ,,

u

v

i

j

I

JJkjIki

v

u

II vu ,

fvu CC ,,

101000000

0

0

C

C

C

v

unew

new

ZYX

vfuf

SVU

vv

uu

fkffkf

Review: Extrinsic Parameters

X

Y

Z C

Image plane

Focal plane

M

m

CCC ZYX ,,

u

v

i

j

I

JJkjIki

v

u

II vu ,

fvu CC ,,Z

Y

X

WWW ZYX ,,

By Rigid Body Transformation:

WC

W

W

W

C

C

C

DMMZYX

TRZYX

110

131

1333

Alper Yilmaz, CAP5415, Fall 2004

8

Estimating Camera Parameters

11, yx 111 ,, ZYX

222 ,, ZYX 333 ,, ZYX

NNN ZYX ,,

22 , yx 33 , yx

NN yx ,

Shape From Images

Perspective cues

Perspective cues

Perspective cues

Ames Room

Ames Room

Video

Recovering 3D from images What cues in the image provide 3D

information?

Shading

Visual cues

Merle Norman Cosmetics, Los Angeles

Visual cues Shading

Texture

The Visual Cliff, by William Vandivert, 1960

Visual cues

From The Art of Photography, Canon

Shading

Texture

Focus

Visual cues Shading

Texture

Focus

Motion

Julesz: had huge impact because it showed that recognition not needed for stereo.

Shape From Multiple Views

Multi-View GeometryRelates

• 3D World Points

• Camera Centers

• Camera Orientations

Multi-View GeometryRelates

• 3D World Points

• Camera Centers

• Camera Orientations

• Camera Intrinsic Parameters

• Image Points

Stereoscene point

optical center

image plane

Stereo

Basic Principle: Triangulation• Gives reconstruction as intersection of two rays

• Requires – calibration– point correspondence

Stereo Constraints

p p’ ?

Given p in left image, where can the corresponding point p’in right image be?

Stereo Constraints

X1

Y1

Z1O1

Image plane

Focal plane

M

p p’Y2

X2

Z2O2

Epipolar Line

Epipole

Epipolar Constraint

From Geometry to Algebra

O O’

P

pp’

From Geometry to Algebra

O O’

P

pp’

Linear Constraint:Should be able to express as matrix multiplication.

The Essential Matrix

Correspondence

Pin Hole Camera Model

ZXfx

Basic Stereo Derivations

Derive expression for Z as a function of x1, x2, f and B

Basic Stereo Derivations

ZXfx 1 Z

BfxZ

BXfx

12

21 xxfBZ

Basic Stereo Derivations

Disparity: 21 xxd

dfBZ

We can always achieve this geometry with image rectification

Image Reprojection reproject image planes onto

common plane parallel to line between optical centers (Seitz)

Rectification example

Correspondence: Epipolar constraint.

Correspondence Problem Two classes of algorithms:

Correlation-based algorithms Produce a DENSE set of correspondences

Feature-based algorithms Produce a SPARSE set of correspondences

Correspondence: Photometric constraint Same world point has same intensity in

both images. Lambertian fronto-parallel Issues:

Noise Specularity Foreshortening

Using these constraints we can use matching for stereo

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image• pick pixel with minimum match cost• This will never work, so:

Improvement: match windows

Comparing Windows: =?

f g

Mostpopular

For each window, match to closest window on epipolar line in other image.

It is closely related to the SSD:

Maximize Cross correlation

Minimize Sum of Squared Differences

Matching cost

disparity

Left Right

scanline

Correspondence search

• Slide a window along the right scanline and compare contents of that window with the reference window in the left image

• Matching cost: SSD or normalized correlation

Left Right

scanline

Correspondence search

SSD

Left Right

scanline

Correspondence search

Norm. corr

Effect of window size

W = 3 W = 20

• Smaller window+ More detail– More noise

• Larger window+ Smoother disparity maps– Less detail– Fails near boundaries

Stereo results

Ground truthScene

Data from University of Tsukuba

(Seitz)

Results with window correlation

Window-based matching(best window size)

Ground truth

(Seitz)

Results with better method

State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

(Seitz)

Failures of correspondence search

Textureless surfaces Occlusions, repetition

Non-Lambertian surfaces, specularities

How can we improve window-based matching?

So far, matches are independent for each point

What constraints or priors can we add?

Stereo constraints/priors• Uniqueness

For any point in one image, there should be at most one matching point in the other image

Stereo constraints/priors• Uniqueness

For any point in one image, there should be at most one matching point in the other image

• Ordering Corresponding points should be in the same order in both views

Ordering constraint doesn’t hold

Priors and constraints

• Uniqueness For any point in one image, there should be at most one

matching point in the other image• Ordering

Corresponding points should be in the same order in both views

• Smoothness We expect disparity values to change slowly (for the

most part)

Stereo matching as energy minimizationI1 I2 D

• Energy functions of this form can be minimized using graph cutsY. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001

W1(i ) W2(i+D(i )) D(i )

)(),;( smooth21data DEIIDEE 2

,neighborssmooth )()(

ji

jDiDE 221data ))(()(

i

iDiWiWE

Examples

bread toy apple

Szeliski

Active stereo with structured light

Project “structured” light patterns onto the object simplifies the correspondence problem

camera 2

camera 1

projector

camera 1

projector

Li Zhang’s one-shot stereo

Active stereo with structured light

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002

Kinect

https://www.youtube.com/watch?v=dTKlNGSH9Po

The third view can be used for verification

Beyond two-view stereo