1

CS 223-BCS 223-BLecture 1Lecture 1

Sebastian Thrun

Gary Bradski

http://robots.stanford.edu/cs223b/index.html

CORNEAAQUEOUSHUMOR

2

Readings

• Computer Vision, Forsyth and Ponce– Chapter 1

• Introductory Techniques for 3D Computer Vision, Trucco and Verri– Chapter 2

3

Lenses and Cameras*

* Slides, where possible, stolen with abandon, many this lecture from Marc Pollefeys comp256, Lect 2

-- Brunelleschi, XVth Century

4

Distant objects appear smaller

A “similar triangle’s” approach to vision. Notes 1.1

Marc Pollefeys

5

Consequences: Parallel lines meet

• There exist vanishing points

Marc Pollefeys

6

Vanishing points

VPL VPRH

VP1VP2

VP3

Different directions correspond to different vanishing points

Marc Pollefeys

7

Implications For Perception*

* A Cartoon Epistemology: http://cns-alumni.bu.edu/~slehar/cartoonepist/cartoonepist.html

Same size things get smaller, we hardly notice…

Parallel lines meet at a point…

8

Implications For Perception 2

Perception must be mapped to a space variant grid

Logrithmic in nature

Steve Lehar

9

The Effect of Perspective

10

Different Projections: Affine projection models: Weak perspective projection

0

'where'

'z

fmmyy

mxx

is the magnification.

When the scene relief is small compared its distance from theCamera, m can be taken constant: weak perspective projection.

Smoosh everything flat onto a parallel plane at distance z0

Marc Pollefeys

11

Affine projection models: Orthographic projection

yy

xx

'

' When the camera is at a(roughly constant) distancefrom the scene, take m=1.

Marc Pollefeys

12

Limits for pinhole cameras

Marc Pollefeys

13

On to Thin Lenses …

Snell’s law

n1 sin 1 = n2 sin 2

Notes 1.2

a b

d e

F

14

Paraxial (or first-order) optics

Snell’s law:

n1 sin 1 = n2 sin 2

Small angles:

n1 1 n22R

nn

d

n

d

n 12

2

2

1

1

R γβα

111

h

d

h

222 R

βγαd

hh

22

11 RR d

hhn

h

d

hn

Sin = y/rTan = y/x Marc Pollefeys

15

Thin Lenses

)1(2 and

11

'

1

n

Rf

fzz

R

n

Z

n

Z

11*

R

n

ZZ

n

1

'

1*

ZR

n

Z

n 11*

'

1111

ZZR

n

R

n

spherical lens surfaces; incoming light parallel to axis; thickness << radii; same refractive index on both sides

'

11* ZR

n

Z

n

R

nn

d

n

d

n 12

2

2

1

1

Notes 1.3 z->Marc Pollefeys

8

16

Thin Lenses summary

)1(2 and

11

'

1 e wher

''

''

n

Rf

fzzz

yzy

z

xzx

http://www.phy.ntnu.edu.tw/java/Lens/lens_e.htmlMarc Pollefeys

17

The depth-of-field

Marc Pollefeys

18

The depth-of-field

fZo

1

Z

1

1

i

iii ZZZ

fZ

Zf

i

i

Zo

yields

d

ZZ

b

Z iii

ii Zbd

bZ

fbdfZ

fZZ ooo /

) ( Z Z Z

0oo

Similar formula for Z Z Z oo o

)( / Z bddZ ii

fZ

ZfZ

o

oi

)(

Z

0o bdfZb

Zdf o

Marc Pollefeys

19

The depth-of-field

fbdfZ

fZZZZZ

/

)(

0

00000

decreases with d, increases with Z0

strike a balance between incoming light and sharp depth range. Notes 1.4

Marc Pollefeys

20

Deviations from the lens model

3 assumptions :

1. all rays from a point are focused onto 1 image point• Remember thin lens small angle assumption

2. all image points in a single plane

3. magnification is constant

Deviations from this ideal are aberrations

0

'

z

fm

Marc Pollefeys

21

Aberrations

chromatic : refractive index function of wavelength

2 types :

1. geometrical

2. chromatic

geometrical : small for paraxial rays

study through 3rd order optics

Marc Pollefeys

22

Geometrical aberrations

spherical aberration

astigmatism

distortion

coma

aberrations are reduced by combining lenses

Marc Pollefeys

23

Spherical aberration

rays parallel to the axis do not converge

outer portions of the lens yield smaller focal lenghts

Marc Pollefeys

24

Astigmatism

Different focal length for inclined rays

Marc Pollefeys

25

Distortion

magnification/focal length different for different angles of inclination

Can be corrected! (if parameters are know)

pincushion(tele-photo)

barrel(wide-angle)

Marc Pollefeys

26

Coma

point off the axis depicted as comet shaped blob

Marc Pollefeys

27

Chromatic aberration

rays of different wavelengths focused in different planes

cannot be removed completely

sometimes achromatization is achieved formore than 2 wavelengths

Marc Pollefeys

28

Vignetting

Marc Pollefeys

29

Calibration Gist: Invert the image formation process

kth collection of points i

Pik

pik

Image plane

x

y

z

0

Cam

era

Rk,Tk

Extrinsic ParamsRotation &Translationto image framecoord. system

f, c, , kIntrinsic Paramsfocuscenter of imageSkew = 0k radial and tangential distortion

(the camera will get several (K) viewsof this grid in rotation)External

coordinatesystem X

Y

Z operator projection theis where),,,,,,( kcfTRPp kkii kk

Note that rotation matrix R has constraints: determinant is 1, inverseis equal to transpose, optimization routine should make use of this.

Then we want the actual projection to be as close as possible toThe point given by the projection operator: over all i pointsand over all k images of grids: kk ii pp

2Argmin],,,,,[

k iiikk kkppkcfTR

• This is typically solved through a gradient decent optimization since the problem is manifestly convex.

• Note that we need a good starting guess for the initial “correct” projection points p’I the optimization then iterates to solution.

• Stereo would then just double the parameters adding left l and right r subscripts and additional summations over r & l.

32

Assumed Perspective Projection

33

Assumed Perspective Projection

34

Cameras

we consider 2 types :

1. CCD

2. CMOS

Marc Pollefeys

35

CCD

separate photo sensor at regular positionsno scanning

charge-coupled devices (CCDs)

area CCDs and linear CCDs2 area architectures : interline transfer and frame transfer

photosensitive

storage

Marc Pollefeys

36

The CCD camera

Marc Pollefeys

37

CMOSSame sensor elements as CCD

Each photo sensor has its own amplifierMore noise (reduced by subtracting ‘black’ image)

Lower sensitivity (lower fill rate)

Uses standard CMOS technologyAllows to put other components on chip

‘Smart’ pixels

Foveon4k x 4k sensor0.18 process70M transistors

Marc Pollefeys

38

CCD vs. CMOS

• Mature technology• Specific technology• High production cost• High power consumption• Higher fill rate• Blooming• Sequential readout

• Recent technology• Standard IC technology• Cheap• Low power• Less sensitive• Per pixel amplification• Random pixel access• Smart pixels• On chip integration

with other components

Marc Pollefeys

39

Colour cameras

We consider 3 concepts:

1. Prism (with 3 sensors)

2. Filter mosaic

3. Filter wheel

… and X3

Marc Pollefeys

40

Prism colour camera

Separate light in 3 beams using dichroic prism

Requires 3 sensors & precise alignment

Good color separation

Marc Pollefeys

41

Prism colour camera

Marc Pollefeys

42

Filter mosaic

Coat filter directly on sensor

Demosaicing (obtain full colour & full resolution image)

Marc Pollefeys

43

Filter wheel

Rotate multiple filters in front of lens

Allows more than 3 colour bands

Only suitable for static scenes

Marc Pollefeys

44

Prism vs. mosaic vs. wheel

Wheel

1

Good

Average

Low

Motion

3 or more

approach

# sensors

Separation

Cost

Frame rate

Artifacts

Bands

Prism

3

High

High

High

Low

3

High-end

cameras

Mosaic

1

Average

Low

High

Aliasing

3

Low-end

cameras

Scientific applications

Marc Pollefeys

45

new color CMOS sensorFoveon’s X3

better image qualitysmarter pixels

Marc Pollefeys

46

The Human Eye

Looking down the optical axis of the eye

Reproduced by permission, the American Society of Photogrammetry andRemote Sensing. A.L. Nowicki, “Stereoscopy.” Manual of Photogrammetry,Thompson, Radlinski, and Speert (eds.), third edition, 1966.

Cross section of the eye

47

Sensors and image processingRGB + B/W happens here

Question: Which way does the light enter?

Light Light

48

Eye cross section

49

The distribution of rods and cones across the retina

Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995). 1995 Sinauer Associates, Inc.

Cones in the fovea

Rods and cones in the periphery

Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995). 1995 Sinauer Associates, Inc.

50

There’s a lot more going on in Vision …i.e. Light and Surfaces

51

Real vision includes invisible inference

52

Real vision includes invisible inference

53

Real vision includes invisible inference