1

Formation et Analyse d’ImagesSession 2

Daniela Hall

7 October 2004

2

Course Overview

• Session 1: – Homogenous coordinates and tensor notation– Image transformations– Camera models

• Session 2:– Camera models– Reflection models– Color spaces

• Session 3:– Review color spaces– Pixel based image analysis– Gaussian filter operators

• Session 4:– Scale Space

3

Course overview

• Session 5:– Contrast description– Hough transform

• Session 6:– Kalman filter– Tracking of regions, pixels, and lines

• Session 7:– Stereo vision – Epipolar geometry

• Session 8: exam

4

Session Overview

1. Camera model

2. Light

3. Reflection models

4. Color spaces

5

Camera model

• Projective model

Ti

Trr

r

Tccc

c

Tsss

s

jiP

yxP

zyxP

zyxP

)1,,(

)1,,(

)1,,,(

)1,,,(

Scene coordinates

Camera coordinates

Image coordinates

6

Camera model

• Transformation from scene to camera coordinates

• Projection of camera coordinates to retina coordinates

• Transformation from retina coordinates to image coordinates

• Composition (camera model) sis

scs

rc

ir

i

rir

i

crc

r

scs

c

PMPTMCP

wPCP

PMP

PTP

The camera model is the composition of the transformations thattransform Ps to Pi

7

Transformation Scene - Camera

• (xs,ys,zs) is position of the origin of the camera system with respect to the scene coordinates (translation).

• R is the orientation of the camera system with respect to the scene system (3d rotation).

1000s

s

scsc

sz

y

x

RT

8

3d rotation

• Around x-axis (counter-clockwise)

• Around y-axis

• Around z-axis

• General )()()(,

1000

1000

0100

00)cos()sin(

00)sin()cos(

)(

1000

0)cos(0)sin(

0010

0)sin(0)cos(

)(

1000

0)cos()sin(0

0)sin()cos(0

0001

)(

aRbRcRRtz

ty

tx

RT

cc

cc

cR

bb

bb

bR

aa

aaaR

xyz

z

y

x

9

Projection Camera-Retina

• Imagine a 1D camera in a 2D space.

• The transformation MRc can be found by considering

similar triangles

z(xc ,zc )

F

x

xr

O

)()(

)()(

c

cr

c

cr

c

cr

c

cr

zF

Fyy

zF

y

F

y

zF

Fxx

zF

x

F

x

10

Projection Camera-Retina

1

0

0

/1

0

0

0

1

0

0

0

1

1

,

F

M

z

y

x

M

w

wy

wx

PMP

RC

c

c

c

RCr

rRR

CR

11

Transformation Retina-Image• A frame: the image is composed of pixels (picture

elements)

• Pixels are in general not squared. There physical sizes depends on the used material.

i columns

j rows

(0,0)

(i-1,j-1)

12

Intrinsic camera parameters

• F: focal distance

• Ci , Cj : Optical image center (in pixels)

• Di , Dj : Physical size of the pixel on the retina (in pixel/mm)

• i, j : image coordinates (in pixels)• Transformation Retina-Image

jjr

iir

CDyj

CDxi

1100

0

0

1r

r

jj

ii

y

x

CD

CD

j

i

13

Camera modelSI

SSC

SRC

IR

I PMPTMCP

1S

S

S

IS z

y

x

M

w

wj

wi

SS

IS

SS

IS

SS

IS

SS

IS

PM

PM

mzsmysmxsm

mzsmysmxsm

w

wjj

PM

PM

mzsmysmxsm

mzsmysmxsm

w

wii

3

2

3

1

)(

)(

34333231

24232221

)(

)(

34333231

14131211

Equation:

Image coordinates

14

Transformation image-scene

• Problem: we need to know depth zs for each image position.

• This process of finding MSI is called

calibration.

• MSI has 12 coefficients.

• MSI is homogenous. 11 degrees of freedom.

ISI

IRI

CR

SC

S PMPCMTP

15

Calibration

1. Construct a calibration object whose 3D position is known.

2. Measure image coordinates

3. Determine correspondences between 3D point RS

k and image point PIk.

4. We have 11 DoF. We need at least 5 ½ correspondences.

16

Calibration

• For each correspondence scene point RSk and image

point PIk

• which gives following equations for k=1, ..., 6

• from wich MIS can be computed

Skh

IS

Skh

IS

k

kkk RM

RM

w

iwi

3

1

)(

)(

Skh

IS

Skh

IS

k

kkk RM

RM

w

jwj

3

2

)(

)(

0))(())((

0))(())((32

31

Skh

ISk

Skh

IS

Skh

ISk

Skh

IS

RMjRM

RMiRM

17

Calibration using many points

• For k=5 ½ M has one solution.– Solution depends on precise measurements of

3D and 2D points. – If you use another 5 ½ points you will get a

different solution.

• A more stable solution is found by using large number of points and do optimisation.

18

Calibration using many points

• For each point correspondence we know (i,j) and R.

• We want to know MIS.

Solve equation with your favorite algorithm (least squares, levenberg-marquart, svd,...)

0))(())((

0))(())((32

31

Skh

ISk

Skh

IS

Skh

ISk

Skh

IS

RMjRM

RMiRM

0

34

33

32

31

24

23

22

21

14

13

12

11

10000

00001

0000000000

0000000000

m

m

m

m

m

m

m

m

m

m

m

m

jzjyjxjzyx

iziyixizyx

19

Application: Rectifying images

20

Applications

21

Homographie: projection from one plane to another

• Homographie HBA is bijective QB = HB

A PA

jj

jjB

jj

jjB

A

j

j

jAB

AB

B

PHPHy

PHPHx

P

H

H

H

PH

w

wy

wx

32

31

3

2

1

/

/

22

Homography computation

• H can be computed from 4 point correspondences.

Ps1 Ps2

Ps3Ps4

Rd1 Rd2

Rd3Rd4Source image (observed)

Destination image (rectified)

4,3,2,1 and 2,1 with ,0 iSRHPRHP DiSD

SiDiSD

Si

23

Homography computation

• H is 3x3 matrix and has 8 degrees of freedom (homogenous coordinates)

• gives 8 equations and one solution for H.

4,3,2,1 and 2,1 with ,0 iSRHPRHP DiSD

SiDiSD

Si

24

Session Overview

1. Camera model

2. Light

3. Reflection models

4. Color spaces

25

Light

• N: surface normal• i angle between incoming light and normal• e angle between normal and camera• g angle between light and camera

camera

light

N

egi

26

Spectrum

• Light source is characterised by its spectrum. • The spectrum consists of a particular quantity of photons

per frequency. • The frequency is described by its wavelength• The visible spectrum is 380nm to 720nm• Cameras can see a larger spectrum depending on their

CCD chip

f1

27

• Albedo is the fraction of light that is reflected by a body or surface.

• Reflectance function:

Albedo

light received

light emitted

Irradiance

Radiance),,( geiR

camera

light

N

egi

28

Session Overview

1. Camera model

2. Light

3. Reflection models

4. Color spaces

29

Reflectance functions

• Specular reflection– example mirror

• Lambertian reflection– diffuse reflection, example paper, snow

30

Specular reflection

else 0,

gei and ei if ,1),,( geiR

camera

lightN

egi

31

Lambertian reflection

)cos(),,( igeiR

32

Di-chromatic reflectance model

• the reflected light R is the sum of the light reflected at the surface Rs and the light reflected from the material body RL

• Rs has the same spectrum as the light source• The spectrum of Rl is « filtered » by the material (photons are

absorbed, this changes the emitted light)• Luminance depends on surface orientation• Spectrum of chrominance is composed of light source

spectrum and absorption of surface material.

),,(),,(),,( geiRgeiRgeiR LLSs

33

Color perception

• The retina is composed of rods and cones.• Rods - provide "scotopic" or low intensity vision.

– Provide our night vision ability for very low illumination, – Are a thousand times more sensitive to light than cones, – Are much slower to respond to light than cones, – Are distributed primarily in the periphery of the visual field.

34

Color perception

• Cones - provide "photopic" or high acuity vision. – Provide our day vision, – Produce high resolution images, – Determine overall brightness or darkness of images, – Provide our color vision, by means of three types of cones:

• "L" or red, long wavelength sensitive, • "M" or green, medium wavelength sensitive, • "S" or blue, short wavelength sensitive.

• Cones enable our day vision and color vision. Rods take over in low illumination. However, rods cannot detect color which is why at night we see in shades of gray.

• source: http://www.hf.faa.gov/Webtraining/VisualDisplays/

35

Color perception

• Rod Sensitivity- Peak at 498 nm.• Cone Sensitivity- Red or "L" cones peak at 564 nm. - Green or "M"

cones peak at 533 nm.  - Blue or "S" cones peak at 437 nm.

• This diagram shows the wavelength sensitivities of the different cones and the rods. Note the overlap in sensitivity between the green and red cones.

36

Camera sensitivity

• observed light intensity depends on:– source spectrum: S(λ)– reflectance of the observed point (i,j): P(i,j,λ)– receptive spectrum of the camera: c(λ)– p0 is the gain

400 600 800 1000 nm

S(λ)

λ

vidicon

CCD

0

0 )()(),,(),( cSjiPpjip

37

Classical RGB camera

• The filters follow a convention of the International Illumination Commission.

• They are functions of λ: r(λ), g(λ), b(λ)• They are close to the sensitivity of the human

color vision system.

38

Color pixels

0

0

0

)()(),,(0),(

)()(),,(0),(

)()(),,(0),(

),(

),(

),(

),(

dbSjiPbjiB

dgSjiPgjiG

drSjiPrjiR

jiB

jiG

jiR

jiP

39

Color bands (channels)

• It is not possible to perceive the spectrum directly.

• Color is a projection of the spectrum to the spectrum of the sensors.

0

0

0

)()(0

)()(0

)()(0

dbSbB

dgSgG

drSrR

40

Session Overview

1. Camera model

2. Light

3. Reflection models

4. Color spaces

41

Color spaces

• RGB color space

• CMY color space

• YIQ color space

• HLS color space

42

RGB color space

• A CCD camera provides RGB images

• The luminance axis is r=g=b (diagonal)

43

CMY color space

• Cyan, magenta, yellow

• CMYK: CMY + black color channel

B

G

R

B

G

R

Y

M

C

max

max

max

44

YIQ color space

• This is an approximation of– Y: luminance, – I: red – cyan, – Q: magenta - green

• Used US TVs (NTSC coding). Black and white TVs display only Y channel.

B

G

R

Q

I

Y

31.052.021.0

32.028.06.0

11.059.03.0

45

HLS space

• Hue, luminance, saturation space.

• L=R+G+B

• S=1-3*min(R,G,B)/L

elsex

gbxT

BGBRGR

BRGRx

,2

if ,

)))(()(

))()((5.0(cos

2

1

L

S T

46

Color distribution

• Color distribution can be studied by histograms.• A histogram is a multi-dimensional table.• We define a function from the continuous color

space to the discrete histogram space.• Then each pixel of the image increments a cell in

the histogram.• Example: We define a histogram of RGB (3D)

with 32 cells/dimension. The pixel value (210,180,100) increments cell (6,5,3)

47

Colors of a surface

• A reflection has the color of the light source (Rs)• Which color is near the border of the reflection?

– Rs and Rb are mixed.

– A color histogram can be used to study this mix.

– The histogram should contain two axis (in theory).

– But reflectance in the real world is more complex than only Rs and Rb. You also have inter reflectance between neighboring objects.