CEE598 - Visual Sensing for
Civil Infrastructure Eng. & Mgmt.
Session 4 – Camera Calibration
Department of Civil and Environmental Engineering, University of Illinois at Urbana-Champaign
Mani Golparvar-Fard Department of Civil and Environmental Engineering
3129D, Newmark Civil Engineering Lab
e-mail: [email protected]
Outline
Camera Calibration
• Review Camera Parameters
• Camera Calibration Problem
• Example
2
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013 Some slides in this lecture are courtesy to Profs. S. Savarese, J.
Ponce & F-F Li
Reading: [FP] Chapter 3
[HZ] Chapter 7
[S] Chapter 6
Pinhole perspective projection f
Oc
f = focal length
Projective camera
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
3
Pinhole perspective projection
x
y
xc
yc
C=[uo, vo]
f
Oc
f = focal length
uo, vo = offset
Projective camera
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
4
f
Oc
Units: k,l [pixel/m]
f [m]
[pixel] , Non-square pixels
f = focal length
uo, vo = offset
non-square pixels ,
Projective camera
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
5
x
y
xc
yc
C=[uo, vo]
f
Oc
f = focal length
uo, vo = offset
non-square pixels , = skewness
Projective camera
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
6
f
Oc
1
z
y
x
0100
0v0
0ucot
P o
o
sin
K has 5 degrees of freedom!
Pc
P’
f = focal length
uo, vo = offset
non-square pixels , = skewness
Projective camera
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
7
f
Oc
Pc
Ow
iw
kw
jw
R,T
wc PTRP
P’
f = focal length
uo, vo = offset
non-square pixels , = skewness
R,T = rotation, translation
Projective camera
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
8
f = focal length
uo, vo = offset
non-square pixels ,
f
Oc
P
Ow
iw
kw
jw
R,T
wPMP
wPTRK
Internal parameters
External parameters
= skewness
R,T = rotation, translation
P’
Projective camera
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
9
wPMP wPTRK
Internal parameters
External parameters
Goal of calibration
Estimate intrinsic and extrinsic parameters
from 1 or multiple images
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
10
wPMP wPTRK
Note: To simplify
notation let P = Pw
100
v0
ucot
K o
o
sin
T
3
T
2
T
1
R
r
r
r
z
y
x
t
t
t
T
Goal of calibration
Estimate intrinsic and extrinsic parameters
from 1 or multiple images
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
11
•P1… Pn with known positions in [Ow,iw,jw,kw]
•p1, … pn known positions in the image
Goal: compute intrinsic and extrinsic parameters
jC
Calibration rig
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
12
jC
Calibration rig
How many correspondences do we need?
•P has 11 unknown • We need 11 equations • 6 correspondences would do it
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
13
image jC
Calibration rig
In practice: user may need to look at the
image and select the n>=6 correspondences
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
14
jC
ii PMP
i
i
iv
up
3
2
1
M
m
m
m
i3
i2
i3
i1
P
P
P
P
m
m
m
m
in pixels
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
15
i3
i1i
P
Pu
m
m
i2i3i P)P(v mm
i1i3i P)P(u mm
i3
i2i
P
Pv
m
m
i
i
v
u
i3
i2
i3
i1
P
P
P
P
m
m
m
m
0Pv
ui
3i2
3i1
mm
mm
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
16
0Pv
ui
3i2
3i1
mm
mm
0Pv
un
3n2
3n1
mm
mm
…
…
0Pv
u1
312
311
mm
mm
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
17
2221
1211
2221
1211
BB
BBB
AA
AAA
What is AB ?
2222122121221121
2212121121121111
BABABABA
BABABABAAB
Review on Block Matrix Multiplication
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
18
2n x 12 12x1
1x4
0Pv
ui
3i2
3i1
mm
mm
T
3
T
2
T
1def
m
m
m
m
4x1
0Pv
un
3n2
3n1
mm
mm
…
…
Homogenous linear system
known unknown
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
19
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Homogeneous M x N Linear Systems
A
A
x
x 0
0 =
=
Square system (M=N):
Rectangular system (M>N)
• 0 is always a solution
• Rank(A) = N
Minimize |Ax|2
under the constraint |x|2 =1
M x N
M=number of equations
N=number of unknown
• noisy measurements
20
How do we solve this homogenous linear system?
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
21
ixix
H
ii xHx
0xHx ii 0Ai h
Function of measurements
unknown
DLT algorithm (Direct Linear Transformation)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
22
1212T
121212n2 VDU
Last column of V gives m
iPM ip
General Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
23
A
T
3
T
2
T
1
A
a
a
a
TRK
3
1
a
3
2
1
b
b
b
b
Estimated values
)(u 21
2
o aa
)(v 32
2
o aa
3231
3231cosaaaa
aaaa
Intrinsic
b
100
v0
ucot
K o
o
sin
Extracting camera parameters
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
24
Theorem (Faugeras, 1993)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
25
A
T
3
T
2
T
1
A
a
a
a
TRK
3
2
1
b
b
b
b
Estimated values
Intrinsic
sin31
2aa
sin32
2aa
b
f
Extracting camera parameters
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
26
A
T
3
T
2
T
1
A
a
a
a
b
TRK
3
2
1
b
b
b
b
Estimated values
Extrinsic
32
321
aa
aar
3
3
1
ar
132 rrr b1KT
Extracting camera parameters
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
27
•Pi’s cannot lie on the same plane!
• Points cannot lie on the intersection curve of two
quadric surfaces
Degenerate cases
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
28
Taking lens distortions into account
Chromatic Aberration
Spherical Aberration
Radial Distortion
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
29
No distortion
Pin cushion
Barrel
Radial Distortion
• Caused by imperfect lenses
• Deviations are most noticeable for rays that pass through the
edge of the lens
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
30
Radial Distortion
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
31
i
i
i
i pv
uPM
100
00
00
1
1
d
v
vucvbuad 222
u
3
1p
2p
pdκ1λ
Polynomial function
Distortion coefficient
To model radial behavior
Radial Distortion
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
32
Estimating m1 and m2…
i
i
iv
up
i3
i2
i3
i1
P
P
P
P
1
m
m
m
m
How to do that?
d
v
u Hint: slopev
u
i
i
Radial Distortion
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
33
Estimating m1 and m2…
i
i
iv
up
i3
i2
i3
i1
P
P
P
P
1
m
m
m
m
0)()( 121111 PuPv mm
0)()( 21 iiii PuPv mm
0)()( 21 nnnn PuPv mm…
0Q n
2
1
m
mn
Tsai technique [87]
i
i
i
i
i
i
i
i
P
P
P
P
P
P
v
u
2
1
3
2
3
1
)(
)(
)(
)(
m
m
m
m
m
m
Radial Distortion
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
34
Once that m1 and m2 are estimated…
i
i
iv
up
i3
i2
i3
i1
P
P
P
P
1
m
m
m
m
3m is non linear function of 1m
2m
There are some degenerate configurations for which m1 and m2 cannot be computed
Radial Distortion
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
35
i
i
iv
up
3
2
1
Q
q
q
q
i3
i2
i3
i1
P
P
P
P
q
q
q
q
i
i
i
i pv
uPM
100
00
00
1
1
Q
PPv
PPu
2i3i
i1i3i
Non-linear system of equations
Radial Distortion
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
36
Radial Distortion
Conversion from distorted coordinates to
undistorted coordinates
• p' = f * r(p) * p
(conversion to pixel coordinates)
r(p) is a function that computes a scaling factor to
undo the radial distortion:
• r(p) = 1.0 + k1 * ||p||^2 + k2 * ||p||^4.
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
37
)(PfX
measurement parameter
f( ) is nonlinear
-Newton Method
-Levenberg-Marquardt Algorithm
• Iterative, starts from initial solution
• May be slow if initial solution far from real solution
• Estimated solution may be function of the initial solution
• Newton requires the computation of J, H
• Levenberg-Marquardt doesn’t require the computation of H
General Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
38
A possible algorithm
1. Solve linear part of the system to find approximated solution
2. Use this solution as initial condition for the full system
3. Solve full system using Newton or L.M.
)(PfX
measurement parameter
f( ) is nonlinear
General Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
39
Typical assumptions:
- zero-skew, square pixel
- uo, vo = known center of the image
- no distortion
Just estimate f
and R, T
)(PfX
measurement parameter
f( ) is nonlinear
General Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
40
Camera Calibration Toolbox for Matlab
J. Bouguet – [1998-2000]
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples
Calibration Procedure
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
41
Calibration Procedure
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
42
Calibration Procedure
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
43
Calibration Procedure
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
44
Calibration Procedure
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
45
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
46
Calibration Procedure
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
47
Calibration Procedure
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
48
Calibration Procedure
Next lecture Single view reconstruction
• Assignment 1 will be online Next Tuesday
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
49
Top Related