#? rahul swaminathan (T-Labs) & professor patrick baudisch

hci2

hasso-plattner institute

determining depth

two subproblems

Matching

Finding corresponding elements in the two images

Reconstruction

Establishing 3-D coordinates from the 2-D image correspondences found during matching

camera

scene

lighting

graphics

light

computer

computer

camera

scene

lighting

light

scene

the camera sees a red pixellet’s assume it correctly classifies it as“glass of red wine”

screen

but, the red wine could be anywhere along this line

computer

two cameras

scene

lighting

light

triangulate the location of the actual glass

wine glass

screens

two subproblems

Matching

Finding corresponding elements in the two images

Reconstruction: done

Establishing 3-D coordinates from the 2-D image correspondences found during matching

two subproblems

Matching: harder

Finding corresponding elements in the two images

Reconstruction: done

Establishing 3-D coordinates from the 2-D image correspondences found during matching

scene

Could we replace one camera with a projector?

<30sec brainstorming>

two cameras

lighting

structured light ::the process of projecting a known pattern of pixels onto a scene

pattern is disturbed when depth changes

gray code

1

Could we achieve the same result with less images?

<30sec brainstorming>

use (cos)-wave pattern instead of b/w

2

pattern needs processingcaveat

Turns out to be a not too hard problem: flood-fill algorithm already provides acceptable solution

continues gradientresult from both

Microsoft Kinect

3

Anoto pen

4

computer

two cameras

scene

lighting

light

main approaches

1.pixel/area-based

2.feature-based

problems

Camera-related problems

- Image noise, differing gain, contrast, etc..

Viewpoint-related problems:

- Perspective distortions

- Occlusions

- Specular reflections

camera positioning

baseline

More matching heuristics

Always valid:

(Epipolar line)

Uniqueness

Minimum/maximum disparity

Sometimes valid:

Ordering

Local continuity (smoothness)

Area-based matching

Finding pixel-to-pixel correspondences

For each pixel in the left image, search for the most similar pixel in the right image

Area-based matching

Finding pixel-to-pixel correspondences

For each pixel in the left image, search for the most similar pixel in the right image

Using neighbourhood windows

Area-based matching

Similarity measures for two windows

SAD (sum of absolute differences)

SSD (sum of squared differences)

CC (cross-correlation)

…

Correspondence via Correlation

Rectified images

Left Right

scanline

SSD error

disparity

(Same as max-correlation / max-cosine for normalized image patch)

Left Disparity Map

Images courtesy of Point Grey Research

Correspondence Using Correlation

Image NormalizationEven when the cameras are identical models, there can be

differences in gain and sensitivity.The cameras do not see exactly the same surfaces, so

their overall light levels can differ.For these reasons and more, it is a good idea to normalize

the pixels in each window:

pixel Normalized ),(

),(ˆ

magnitude Window )],([

pixel Average ),(

),(

),(),(

2

),(

),(),(),(

1

yxW

yxWvuyxW

yxWvuyxW

m

mm

m

m

II

IyxIyxI

vuII

vuII

problems

problems

Scale change

Rotation

Occlusion

Illumination

……

result

combining both

wine glass

screens

the underlying problem is:compute the intersection of two lines

commonly compute “depth” image

end