Lecture 2 Camera Calibration

70
1 Lecture 2: Camera Calibration Lecture 2 Camera Calibration Joaquim Salvi Universitat de Girona Visual Perception

Transcript of Lecture 2 Camera Calibration

Page 1: Lecture 2   Camera Calibration

1

Lecture 2: Camera Calibration

Lecture 2Camera Calibration

Joaquim SalviUniversitat de Girona

Visual Perception

Page 2: Lecture 2   Camera Calibration

2

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 3: Lecture 2   Camera Calibration

3

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 4: Lecture 2   Camera Calibration

4

Lecture 2: Camera Calibration

– Dense reconstruction – Visual inspection

– Object localization – Camera localization

2.1 Calibration Introduction

• Some applications of this capability include

Page 5: Lecture 2   Camera Calibration

5

Lecture 2: Camera Calibration

Image courtesy of C. Taylor

“The Scholar of Athens,” Raphael, 1518

2.1 Calibration Introduction – Perspective Imaging

Page 6: Lecture 2   Camera Calibration

6

Lecture 2: Camera Calibration

WZ

WY

WX

WO

wP

Image Plane

{ }W

uP

IY

IX

IO

{ }I

Focal Point

WZ

WY

WX

WO

wP

Image Plane

{ }W

uP

IY

IX

IO

{ }I

Focal Point

1

I

u

I I

u u

X

P Y

1

W

w

W

W w

w W

w

X

YP

Z

In pixels

In metrics?

wl

wl

0

W

w

W

W w

w W

w

X

Yl

Z

2.1 Calibration Introduction

Page 7: Lecture 2   Camera Calibration

7

Lecture 2: Camera Calibration

Modelling

G(X) X ?

Calibration

X !!!

Modelling:

• Determine the equation that approximates the camera behaviour.

• Define the set of unknowns in the equation (camera parameters).

• The camera model is an approximation of the physics & optics of the camera.

Calibration:

• Get the numeric value of every camera parameter.

G(X)

2.1 Calibration Introduction

Page 8: Lecture 2   Camera Calibration

8

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 9: Lecture 2   Camera Calibration

9

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 10: Lecture 2   Camera Calibration

10

Lecture 2: Camera Calibration

2.2 Pinhole Model

Camera

coordinate

system

World

coordinate

system

0 0,u v

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

dP

IY

IXIO{ }I

RY

RX{ }R

RO

Image

coordinate

system

Page 11: Lecture 2   Camera Calibration

11

Lecture 2: Camera Calibration

2.2 Pinhole Model (Step 1: World to Camera)

Camera

coordinate

system

World

coordinate

system

CY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

C

WK

Step 1

Page 12: Lecture 2   Camera Calibration

12

Lecture 2: Camera Calibration

2.2 Pinhole Model (Step 2: Projection)

Camera

coordinate

system

World

coordinate

system

CY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uPf

Step 2

wX

wY

wZ

uX

uY

RY

RX{ }R

RO

Page 13: Lecture 2   Camera Calibration

13

Lecture 2: Camera Calibration

2.2 Pinhole Model (Step 3: Lens Distortion)

Camera

coordinate

system

World

coordinate

system

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

RY

RX{ }R

RO

Step 3

dP

Page 14: Lecture 2   Camera Calibration

14

Lecture 2: Camera Calibration

2.2 Pinhole Model (Step 3: Lens Distortion)

dP

uP dr

CY

CX

Observed position

Ideal

projection

dr: radial distortion

a

b

Radial distortion effect (a: negative, b: positive)

Radial Distortion

Page 15: Lecture 2   Camera Calibration

15

Lecture 2: Camera Calibration

2.2 Pinhole Model (Step 3: Lens Distortion)

Axis with

maximum

radial

distortion

Axis with

minimum

tangential

distortion

CY

CX

Ideal

projection

Observed

position

dr: radial distortion

dt: tangential distortion

dPuP

dr

CY

CX

dt

Radial and Tangential Distortion

Image with distortionImage without distortion

Page 16: Lecture 2   Camera Calibration

16

Lecture 2: Camera Calibration

2.2 Pinhole Model (Step 4: Camera to Image)

Camera

coordinate

system

World

coordinate

system

0 0,u v

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

IY

IXIO{ }I

RY

RX{ }R

RO

Image

coordinate

system

Step 4

dP

Page 17: Lecture 2   Camera Calibration

17

Lecture 2: Camera Calibration

Camera

coordinate

system

World

coordinate

system

0 0,u v

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

dP

IY

IXIO{ }I

RY

RX { }R

RO

C

WK

Image

coordinate

system

Step 1

Step 2Step 3

Step 4

2.2 Pinhole Model

Page 18: Lecture 2   Camera Calibration

18

Lecture 2: Camera Calibration

2.2 Calibration Methods (I)

• Method of Hall– Lineal method

– Transformation matrix

• Method of Faugeras-Toscani– Lineal method

– Obtaining camera parameters

• Method of Faugeras-Toscani with distortion– Iterative method

– Radial distortion

• Method of Tsai– Iterative method

– Radial distortion

– Focal distance estimation

• Method of Weng– Iterative method

– Radial and tangential distortion

• … and many more

Page 19: Lecture 2   Camera Calibration

19

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 20: Lecture 2   Camera Calibration

20

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 21: Lecture 2   Camera Calibration

21

Lecture 2: Camera Calibration

2.3 The Method of Hall

• Method of Hall– Lineal method

– Transformation matrix

• Method of Faugeras-Toscani– Lineal method

– Obtaining camera parameters

• Method of Faugeras-Toscani with distortion– Iterative method

– Radial distortion

• Method of Tsai– Iterative method

– Radial distortion

– Focal distance estimation

• Method of Weng– Iterative method

– Radial and tangential distortion

• … and many more

Page 22: Lecture 2   Camera Calibration

22

Lecture 2: Camera Calibration

World

coordinate

system

WZ

WYWX

WO

wP

Image plane

{ }W

uP

IY

IXIO{ }IImage

coordinate

system

2.3 The Method of Hall - Modelling

Page 23: Lecture 2   Camera Calibration

23

Lecture 2: Camera Calibration

11 12 13 14

21 22 23 24

31 32 33 341

W

I w

u W

I w

u W

w

Xs X A A A A

Ys Y A A A A

Zs A A A A

Assume light is captured on the image plane by a linear projection

The matrix is defined up to a scale factor Multiple Solutions

A component is fixed to the unity Unique Solution

11 12 13 14

21 22 23 24

31 32 33 11

W

I w

u W

I w

u W

w

Xs X A A A A

Ys Y A A A A

Zs A A A

2.3 The Method of Hall - Modelling

Page 24: Lecture 2   Camera Calibration

24

Lecture 2: Camera Calibration

11 12 13 14

31 32 33

21 22 23 24

31 32 33

1

1

W W W

I w w w

u W W W

w w w

W W W

I w w w

u W W W

w w w

A X A Y A Z AX

A X A Y A Z

A X A Y A Z AY

A X A Y A Z

11 12 13 14

21 22 23 24

31 32 33 11

W

I w

u W

I w

u W

w

Xs X A A A A

Ys Y A A A A

Zs A A A

11 31 12 32 13 33 14

21 31 22 32 23 33 24

W I W W I W W I W I

w u w w u w w u w u

W I W W I W W I W I

w u w w u w w u w u

A X A X X A Y A X Y A Z A X Z A X

A X A Y X A Y A Y Y A Z A Y Z A Y

2.3 The Method of Hall - Calibration

Page 25: Lecture 2   Camera Calibration

25

Lecture 2: Camera Calibration

2 1

2

1 0 0 0 0

0 0 0 0 1

W W W I W I W I W

i w w w u w u w u wi i i i i i i i i

W W W I W I W I W

i w w w u w u w u wi i i i i i i i i

Q X Y Z X X X Y X Z

Q X Y Z Y X Y Y Y Z

2 1

2

I

i u i

I

i u i

B X

B Y

T

11 12 13 14 21 22 23 24 31 32 33A A A A A A A A A A A A

1

t tA Q Q Q B

QA B

Pseudoinverse leads to a unique solution:

1A Q B

Obtaining 11 unknowns and each 2D point gives two equations

So, at least 6 points are needed. More points leads to a more accurate solution.

11 31 12 32 13 33 14

21 31 22 32 23 33 24

W I W W I W W I W I

w u w w u w w u w u

W I W W I W W I W I

w u w w u w w u w u

A X A X X A Y A X Y A Z A X Z A X

A X A Y X A Y A Y Y A Z A Y Z A Y

2.3 The Method of Hall - Calibration

Page 26: Lecture 2   Camera Calibration

26

Lecture 2: Camera Calibration

Camera

coordinate

system

CY

CX CZ

CO{ }C

IY

IXIO{ }I

RY

RX

RO

{ }R

World

coordinate

system

WZ

WYWX

WO{ }W

Reconstruction

Area

Image of the calibrating pattern

2 1

2

1 0 0 0 0

0 0 0 0 1

W W W I W I W I W

i w w w u w u w u wi i i i i i i i i

W W W I W I W I W

i w w w u w u w u wi i i i i i i i i

Q X Y Z X X X Y X Z

Q X Y Z Y X Y Y Y Z

2 1

2

I

i u i

I

i u i

B X

B Y

1t t

A Q Q Q B

2.3 The Method of Hall - Calibration

Page 27: Lecture 2   Camera Calibration

27

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 28: Lecture 2   Camera Calibration

28

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 29: Lecture 2   Camera Calibration

29

Lecture 2: Camera Calibration

2.4 The Method of Faugeras-Toscani

• Method of Hall– Lineal method

– Transformation matrix

• Method of Faugeras-Toscani– Lineal method

– Obtaining camera parameters

• Method of Faugeras-Toscani with distortion– Iterative method

– Radial distortion

• Method of Tsai– Iterative method

– Radial distortion

– Focal distance estimation

• Method of Weng– Iterative method

– Radial and tangential distortion

• … and many more

Page 30: Lecture 2   Camera Calibration

30

Lecture 2: Camera Calibration

Camera

coordinate

system

World

coordinate

system

0 0,u v

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

IY

IXIO{ }I

RY

RX { }R

RO

C

WK

Image

coordinate

system

Step 1

Step 2Step 3

Step 4

2.4 The Method of Faugeras-Toscani

Page 31: Lecture 2   Camera Calibration

31

Lecture 2: Camera Calibration

• Extrinsic parameters: Model the situation and orientation of the camera with

respect to a world co-ordinate system.

• Intrinsic parameters: Model the behaviour of the internal geometry and the optical

characteristics of the camera.

u

w

v

Yc

Xc

Zc

Oc

Oi

(u0, v0)

Pu

P

image

co-ordinate

system

(píxels)

retinal

co-ordinate

system

(mm.)

Image plane

Retinal plane

Yr

Xr

Zr

w

World

co-ordinate

system

WZ

WY

WX

WO { }W

Camera

co-ordinate system

2.4 The Method of Faugeras-Toscani

Page 32: Lecture 2   Camera Calibration

32

Lecture 2: Camera Calibration

Yc

Xc

Zc

Zw

YwOc

Ow

Pw

Camera

co-ordinate

system World

co-ordinate system

Retinal Plane

K

Xw

X

C

W Y

Z

t

T t

t

11 12 13

21 22 23

31 32 33

, , ,C

W

C

W

R Rot X Rot Y Rot Z

r r r

R r r r

r r r

C W

w w

C C W C

w W w W

C W

w w

X X

Y R Y T

Z Z

1 1

C W

Cw w

W

P PK

3 3 3 1

1 30 1

C C

C W Wx x

W

x

R TK

2.4 Extrinsic Parameters

Page 33: Lecture 2   Camera Calibration

33

Lecture 2: Camera Calibration

CPw

CPu

Yc

Xc

Zc

Oc C

f

PZc

Yu

PYc XuPXc

C

C w

u C

w

C

C w

u C

w

XX f

Z

YY f

Z

2.4 The Intrinsic Parameters: Ideal Projection

Page 34: Lecture 2   Camera Calibration

34

Lecture 2: Camera Calibration

pixel

Retinal

plane

(0, 0)

Yr

Xr (0, 0)

(Xd, Yd)

Image

Plane

(Xp, Yp)

R C

d u u

R C

d v u

X k X

Y k Y

2.4 The Intrinsic Parameters: Pixel Conversion

Page 35: Lecture 2   Camera Calibration

35

Lecture 2: Camera Calibration

Yr

Xr

V

U(0, 0)

Principal point

(u0,v0)

Computer image

co-ordinate

system

Camera

co-ordinate

system

0

0

I R

d d

I R

d d

X X u

Y Y v

2.4 The Intrinsic Parameters: Principal Point

Page 36: Lecture 2   Camera Calibration

36

Lecture 2: Camera Calibration

Camera

coordinate

system

World

coordinate

system

0 0,u v

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

IY

IXIO{ }I

RY

RX { }R

RO

C

WK

Image

coordinate

system

Step 1

Step 2Step 3

Step 4

2.4 The Method of Faugeras-Toscani

Page 37: Lecture 2   Camera Calibration

37

Lecture 2: Camera Calibration

Real projection on the image plane (Xi, Yi)

(Xw, Yw, Zw) 3D object point with respect to world co-ordinate system

Affine transformation.

Modelled parameters: R, T

(Xc, Yc, Zc) 3D object point with respect to camera co-ordinate system

Perspective transformation.

Modelled parameter: f

(Xu, Yu) Ideal projection on the retinal plane

Pixel adjustment

Modelled parameters: ku, kv

(Xp, Yp) Real projection on the image plane

Adaptation to the computer image buffer

Modelled parameters: u0, v0

2.4 The Method of Faugeras-Toscani

Page 38: Lecture 2   Camera Calibration

38

Lecture 2: Camera Calibration

C

C w

u C

w

C

C w

u C

w

XX f

Z

YY f

Z

R C

d u u

R C

d v u

X k X

Y k Y

0

0

I R

d d

I R

d d

X X u

Y Y v

0

0

C

I w

u u C

w

C

I w

u v C

w

XX k f u

Z

YY k f v

Z

0

0

0 0

0 0

0 0 1 01

C

I w

u u C

I w

u v C

w

Xs X u

Ys Y v

Zs

vv

uu

fk

fk

2.4 The Method of Faugeras-Toscani - Modelling

Page 39: Lecture 2   Camera Calibration

39

Lecture 2: Camera Calibration

11 12 13

0

21 22 23

0

31 32 33

0 0

0 0

0 0 1 00 0 0 1 1

W

xI w

u u W

yI w

u v W

z w

r r r t Xs X u

r r r t Ys Y v

r r r t Zs

1 0 3 0

2 0 3 0

3

u u x z

v v y z

z

r u r t u t

A r v r t v t

r t

Intrínsecs Extrínsecs

2.4 The Method of Faugeras-Toscani - Modelling

Page 40: Lecture 2   Camera Calibration

40

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 41: Lecture 2   Camera Calibration

41

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 42: Lecture 2   Camera Calibration

42

Lecture 2: Camera Calibration

11 12 13

0

21 22 23

0

31 32 33

0 0

0 0

0 0 1 00 0 0 1 1

W

xI w

u u W

yI w

u v W

z w

r r r t Xs X u

r r r t Ys Y v

r r r t Zs

1 0 3 0

2 0 3 0

3

u u x z

v v y z

z

r u r t u t

A r v r t v t

r t

Intrínsecs Extrínsecs

2.5 The Method of Faugeras-Toscani – Modelling

Page 43: Lecture 2   Camera Calibration

43

Lecture 2: Camera Calibration

1 14 3 34

2 24 3 34

0

0

W I W

w u w

W I W

w u w

A P A X A P A

A P A Y A P A

31 14

34 34 34

32 24

34 34 34

I W W I

u w w u

I W W I

u w w u

AA AX P P X

A A A

AA AY P P Y

A A A

1 1 2

3 2 2

I W W I

u w w u

I W W I

u w w u

X T P C T P X

Y T P C T P Y

v

z

y

v

zz

z

u

z

xu

zz

t

tvC

t

rv

t

rT

t

rT

t

tuC

t

ru

t

rT

022

03

3

32

011

03

1

2.5 The Method of Faugeras-Toscani – Calibration

1 0 3 0

2 0 3 0

3

u u x z

v v y z

z

r u r t u t

A r v r t v t

r t

343

242

141

AA

AA

AA

1343

242

141

w

W

u

I

u

I

P

AA

AA

AA

s

Ys

Xs

Page 44: Lecture 2   Camera Calibration

44

Lecture 2: Camera Calibration

1

2

3

1

2

T

T

X T

C

C

B QX

1 3

1 3

0 1 0

0 0 1

t tW I W

w u w xi i i

t tI W W

x u w wi i i

P X PQ

Y P P

I

u i

I

u i

XB

Y

1

t tX Q Q Q B

2.5 The Method of Faugeras-Toscani – Calibration

1 1 2

3 2 2

I W W I

u w w u

I W W I

u w w u

X T P C T P X

Y T P C T P Y

11 unknowns

minimum 6 points

Page 45: Lecture 2   Camera Calibration

45

Lecture 2: Camera Calibration

v

z

y

v

zz

z

u

z

xu

zz

t

tvC

t

rv

t

rT

t

rT

t

tuC

t

ru

t

rT

022

03

3

32

011

03

1

3 1r 2

1zt

T

2.5 The Method of Faugeras-Toscani – tz

𝑅 =

𝑟1𝑟2𝑟3

Page 46: Lecture 2   Camera Calibration

46

Lecture 2: Camera Calibration

1 2 1 2

1 2 1 2

cos

sin

v v v v

v v v v

0

1

1

0

t

i j

t

i j

i j

i j

r r i j

r r i j

r r i j

r r i j

v

z

y

v

zz

z

u

z

xu

zz

t

tvC

t

rv

t

rT

t

rT

t

tuC

t

ru

t

rT

022

03

3

32

011

03

1

2.5 The Method of Faugeras-Toscani – Intrinsics

02

33130

33310

32121

·· u

t

rr

t

r

t

ru

t

r

t

r

t

r

t

ru

t

rTTTT

z

u

zzzzz

u

zz

t

2

1zt

T

2

2

210

T

TTu

t

Page 47: Lecture 2   Camera Calibration

47

Lecture 2: Camera Calibration

1 2 1 2

1 2 1 2

cos

sin

v v v v

v v v v

0

1

1

0

t

i j

t

i j

i j

i j

r r i j

r r i j

r r i j

r r i j

v

z

y

v

zz

z

u

z

xu

zz

t

tvC

t

rv

t

rT

t

rT

t

tuC

t

ru

t

rT

022

03

3

32

011

03

1

2 31 2

0 02 2

2 2

1 2 2 3

2 2

2 2

tt

t t t t

u v

T TT Tu v

T T

T T T T

T T

2.5 The Method of Faugeras-Toscani – Intrinsics

Page 48: Lecture 2   Camera Calibration

48

Lecture 2: Camera Calibration

1 2 1 2

1 2 1 2

cos

sin

v v v v

v v v v

0

1

1

0

t

i j

t

i j

i j

i j

r r i j

r r i j

r r i j

r r i j

v

z

y

v

zz

z

u

z

xu

zz

t

tvC

t

rv

t

rT

t

rT

t

tuC

t

ru

t

rT

022

03

3

32

011

03

1

2.5 The Method of Faugeras-Toscani – Extrinsics

tt

t

tt

t

u

z

z

u

zz

TT

T

T

TTTTr

TTT

T

T

TTTTr

tu

t

rTr

t

ru

t

rT

21

2

2

2

21211

221

2

2

2

2

212110

311

10

31

1

Page 49: Lecture 2   Camera Calibration

49

Lecture 2: Camera Calibration

1 2 1 2

1 2 1 2

cos

sin

v v v v

v v v v

0

1

1

0

t

i j

t

i j

i j

i j

r r i j

r r i j

r r i j

r r i j

v

z

y

v

zz

z

u

z

xu

zz

t

tvC

t

rv

t

rT

t

rT

t

tuC

t

ru

t

rT

022

03

3

32

011

03

1

2 1 2

1 1 22

1 2 2

2 2 3

2 3 22

2 3 2

2

3

2

t

t t

t

t t

T T Tr T T

T T T

T T Tr T T

T T T

Tr

T

2 1 2

1 2

1 2 2

2 2 3

2 2

2 3 2

2

1

t

x t t

t

y t t

z

T T Tt C

T T T

T T Tt C

T T T

tT

2.5 The Method of Faugeras-Toscani – Extrinsics

Page 50: Lecture 2   Camera Calibration

50

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 51: Lecture 2   Camera Calibration

51

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 52: Lecture 2   Camera Calibration

52

Lecture 2: Camera Calibration

2.6 The Method of Faugeras-Toscani with distortion

• Method of Hall– Lineal method

– Transformation matrix

• Method of Faugeras-Toscani– Lineal method

– Obtaining camera parameters

• Method of Faugeras-Toscani with distortion– Iterative method

– Radial distortion

• Method of Tsai– Iterative method

– Radial distortion

– Focal distance estimation

• Method of Weng– Iterative method

– Radial and tangential distortion

• … and many more

Page 53: Lecture 2   Camera Calibration

53

Lecture 2: Camera Calibration

Camera

coordinate

system

World

coordinate

system

0 0,u v

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

dP

IY

IXIO{ }I

RY

RX { }R

RO

C

WK

Image

coordinate

system

Step 1

Step 2Step 3

Step 5

Step 4

2.6 The Method of Faugeras-Toscani with distortion

Page 54: Lecture 2   Camera Calibration

54

Lecture 2: Camera Calibration

Ideal

projection

Observed

position

drdt

Xr

Yr

dr: radial distortion

dt: tangential distortion

PuPd

2.6 Lens Distortion

Page 55: Lecture 2   Camera Calibration

55

Lecture 2: Camera Calibration

a

b

Radial distorsion effect Tangential distorsion effect

Xr

Axe of a

maximum

tangential

distortion

Axe of a

minimum

tangential

distortion

Radial distorsion is the most important and usually the only considered in

calibration.

2.6 Lens Distortion

Page 56: Lecture 2   Camera Calibration

56

Lecture 2: Camera Calibration

X X Du d x Y Y Du d y

D X k rx d 1

2 D Y k ry d 1

2r X Yd d 2 2

2 4

1 2

2 4

1 2

2 2

C

x d

C

y d

C C

d d

D X k r k r

D Y k r k r

r X Y

k1 is the most important component

and usuallly sufficient in most

applications.

2.6 Lens Distortion

Model of Faugeras-Toscani with distortion:

Page 57: Lecture 2   Camera Calibration

57

Lecture 2: Camera Calibration

u

w

v

Yc

Xc

Zc

Oc

Oi

(u0, v0)

Pu

P

Camera

co-ordinate system

image

co-ordinate

system

f Pd

retinal

co-ordinate

system

Image plane

Retinal plane

Yr

Xr

Zr

X

f

P

P

u Xc

Zc

Y

f

P

P

u Yc

Zc

X X Du d x Y Y Du d y

D X k rx d 1

2 D Y k ry d 1

2r X Yd d 2 2

X k Xp u d Y k Yp v d

X X ui p 0 Y Y vi p 0

2.6 The Method of Faugeras-Toscani with distortion

Page 58: Lecture 2   Camera Calibration

58

Lecture 2: Camera Calibration

Camera

coordinate

system

World

coordinate

system

0 0,u v

fCY

CX CZ

CO

WZ

WYWX

WO

wP

Image plane

{ }W

{ }C

uP

dP

IY

IXIO{ }I

RY

RX { }R

RO

C

WK

Image

coordinate

system

Step 1

Step 2Step 3

Step 5

Step 4

2.6 The Method of Faugeras-Toscani with distortion

Page 59: Lecture 2   Camera Calibration

59

Lecture 2: Camera Calibration

(Xw, Yw, Zw) 3D object point with respect to world co-ordinate system

Affine transformation.

Modelled parameters: R, T

(Xc, Yc, Zc) 3D object point with respect to camera co-ordinate system

Perspective transformation.

Modelled parameter: f

(Xu, Yu) Ideal projection on the retinal plane

Radial lens distortion.

Modelled parameter: k1

(Xd, Yd) Real projection on the retinal plane

Pixel adjustment

Modelled parameters: ku, kv

(Xp, Yp ) Real projection on the image plane

Adaptation to the computer image buffer

Modelled parameters: u0, v0

(Xi, Yi) Real projection on the image plane

2.6 The Method of Faugeras-Toscani with distortion

Page 60: Lecture 2   Camera Calibration

60

Lecture 2: Camera Calibration

2

1

2

1

C

C Cw

d dC

w

C

C Cw

d dC

w

Xf X k r X

Z

Yf Y k r Y

Z

0

0

I

dC

d

u

I

dC

d

v

X uX

k

Y vY

k

1 1

C W

w w

C W

Cw w

WC W

w w

X X

Y YK

Z Z

r X Yd d 2 2

The model is NON-LINEARIterative minimisation:

• Newton-Raphson

• Levenberg-Marquardt

2.6 The Method of Faugeras-Toscani with distortion

Page 61: Lecture 2   Camera Calibration

61

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 62: Lecture 2   Camera Calibration

62

Lecture 2: Camera Calibration

Contents

2. Camera Calibration2.1 Calibration introduction

2.2 The pinhole model

2.3 The method of Hall

2.4 The method of Faugeras-Toscani – Modelling

2.5 The method of Faugeras-Toscani – Calibration

2.6 The method of Faugeras-Toscani with distortion

2.7 Experimental comparison of methods

Page 63: Lecture 2   Camera Calibration

63

Lecture 2: Camera Calibration

Hall Faugeras Faugeras distorted

Tsai Weng

Transformation

matrix

Step 3Lens

Distortion

Step 2Projection

Step 1World2camera

Transformation

with , , , tx, ty and tz

Projection with f

Radial distortion with k1

UndistortedMultiple

distortion

k1, g1, g2, g3, g4

Transformation with

u0, v0 , ku and kv

Transformation

with u0, v0 and sx Transformation

with u0, v0,

ku and kv

Step 4Camera2image

C C W C

w W w WP P T R

,

C C

C Cw w

u uC C

w w

X YX f Y f

Z Z

C C

w uP P

C C

u dP P

C I

d dP P

C C

u dP P

0

0

I C

d u d

I C

d v d

X k X u

Y k Y v

2 2

1

2 2

1

C C C C C

u d u u u

C C C C C

u d u u u

X X k X X Y

Y Y k Y X Y

1'

0

1

0

I C

d x x d

I C

d y d

X s d X u

Y d Y v

=

W C

w wP P

I W

d wP P A

2.7 Experimental Comparison - Methods

Page 64: Lecture 2   Camera Calibration

64

Lecture 2: Camera Calibration

wP

Optical Ray

3Dd

2.7 Experimental Comparison - Accuracy Evaluation

• 3D Measurement

– Distance with respect to the optical ray

– Normalized Stereo Calibration Error

• 2D Measurement

– Accuracy of distorted image coordinates

– Accuracy of undistorted image

coordinates

1 22 2

2 2 21

ˆ ˆ1

NSCEˆ 12

C C C Cn

w w w wi i i i

Ci w u vi

X X Y Y

n Z

Camera

coordinate

system

World

coordinate

system

fCY

CX

WZ

WX

WO

wP

Image plane

{ }W

uP

dP

IX{ }I

RX

{ }R

RO

Image

coordinate

system

ˆuP

ˆdP

0 0,u v

WYCZ

CO { }C

IY

IO

RY

uP

ˆuP

0 0,u v

dd

Observed Point

Linear Projection

- distortion

+ distortion

ud

ˆdP

dP

Page 65: Lecture 2   Camera Calibration

65

Lecture 2: Camera Calibration

2.7 Experimental Comparison: Synthetic Images (I)

2D distorted image (pix.) 2D undistorted image (pix.)

Mean Standard

desviation

Max Min Mean Standard

desviation

Max Min

1 Hall 0.2676 0.1979 1.2701 0.0213 0.2676 0.1979 1.2701 0.0213

2 Faugeras 0.2689 0.1997 1.2377 0.0075 0.2689 0.1997 1.2377 0.0075

3 Faugeras with distortion 0.0840 0.0458 0.2603 0.0081 0.0834 0.0454 0.2561 0.0080

4 Tsai 0.0838 0.0457 0.2426 0.0035 0.0832 0.0453 0.2386 0.0035

5 Weng 0.0845 0.0455 0.2608 0.0019 0.0843 0.0443 0.2584 0.0129

2D distorted

0

0,05

0,1

0,15

0,2

0,25

0,3

1 2 3 4 5

pix

.

Mean Standard deviation

2D undistorted

0

0,05

0,1

0,15

0,2

0,25

0,3

1 2 3 4 5p

ix.

Mean Standard deviation

Page 66: Lecture 2   Camera Calibration

66

Lecture 2: Camera Calibration

2.7 Experimental Comparison: Synthetic Images (II)

3D position (mm) NSCE

Mean Standard

desviation

Max Min

1 Hall 0.1615 0.1028 0.5634 0.0113 n/a

2 Faugeras 0.1811 0.1357 0.8707 0.0147 0.6555

3 Faugeras NR with distortion 0.0566 0.0307 0.1694 0.0055 0.2042

4 Tsai optimized 0.0565 0.0306 0.1578 0.0087 0.2037

5 Weng 0.0570 0.0305 0.1696 0.0088 0.2064

Normalized Stereo Calibration Error

Normalized Stereo Calibration Error

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

1 2 3 4 5

NSCE

3D position

0

0,05

0,1

0,15

0,2

1 2 3 4 5

mm

.

Mean Standard deviation

Page 67: Lecture 2   Camera Calibration

67

Lecture 2: Camera Calibration

Computing Time

160 punts 1800 punts

• Hall 1 ms 70 ms

• Faugeras 1 ms 70 ms

• Faugeras with distortion 10 ms 380 ms

• Tsai 10 ms 530 ms

• Weng 51 ms 4216 ms

Pentium III at 1 GHz.

2.7 Experimental Comparison: Synthetic Images (III)

Page 68: Lecture 2   Camera Calibration

68

Lecture 2: Camera Calibration

2.7 Experimental Comparison: Real Images (I)

Camera

coordinate

system

CY

CX CZ

CO{ }C

IY

IXIO{ }I

RY

RX

RO

{ }R

World

coordinate

system

WZ

WYWX

WO{ }W

Reconstruction

Area

Image of the calibrating pattern

3D position (mm) NSCE

Mean Standard

desviation

Max Min

Hall 0.5219 0.2595 1.1370 0.0143 n/a

Faugeras 0.7782 0.4253 2.0210 0.0187 4.0649

Faugeras with distortion 0.4967 0.3367 1.5642 0.0094 2.5489

Tsai 0.4815 0.3023 1.4014 0.0093 2.4836

Weng 0.4740 0.2904 1.2669 0.0087 2.4556

Page 69: Lecture 2   Camera Calibration

69

Lecture 2: Camera Calibration

2.7 Experimental Comparison: Real Images (II)

3D position (mm) NSCE

Mean Standard

desviation

Max Min

Hall 1.5698 0.9842 8.9249 0.0247 n/a

Faugeras 1.6187 0.9856 8.8812 0.0302 2.0175

Faugeras with distortion 0.9930 0.5660 3.2386 0.0154 0.9909

Tsai 0.9927 0.5655 3.2311 0.0153 0.9908

Weng 0.9896 0.5724 3.3526 0.0149 0.9869

Image of the calibration patternStereo camera over a mobile robot

Page 70: Lecture 2   Camera Calibration

70

Lecture 2: Camera Calibration

2.7 Experimental Comparison - Conclusions

• Implementation of 5 of the most used camera calibration

methods

– Notation was unified

– The methods were compared in terms of model and

calibration

• The accuracy of non-linear methods is better than linear

methods

• Modelling of radial distortion is quite sufficient when high

accuracy is required

• Accuracy measuring methods obtain similar results if they are

relatively compared

Additional bibliography:

J. Salvi, X. Armangué and J. Batlle. A Comparative Review of Camera Calibrating Methods with Accuracy Evaluation. Pattern Recognition, PR, pp. 1617-1635, Vol. 35, Issue 7, July 2002.