Post on 11-Jul-2020
Lecture 8 -Fei-Fei Li
Lecture 8:
Camera Calibration
Professor Fei-Fei Li
Stanford Vision Lab
19-Oct-111
Lecture 8 -Fei-Fei Li
What we will learn today?
• Review camera parameters
• Affine camera model (Problem Set 2 (Q4))
• Camera calibration
• Vanishing points and lines (Problem Set 2
(Q1))
19-Oct-112
Reading:
• [FP] Chapter 3
• [HZ] Chapter 7, 8.6
Lecture 8 -Fei-Fei Li
What we will learn today?
• Review camera parameters
• Affine camera model
• Camera calibration
• Vanishing points and lines
19-Oct-113
Reading:
• [FP] Chapter 3
• [HZ] Chapter 7, 8.6
Lecture 8 -Fei-Fei Li 19-Oct-114
Projective cameraf
Oc
f = focal length
Lecture 8 -Fei-Fei Li 19-Oct-115
Projective camera
x
y
xc
yc
C=[uo, vo]
f
Oc
f = focal length
uo, vo = offset
Lecture 8 -Fei-Fei Li 19-Oct-116
Projective cameraf
Oc
Units: k,l [pixel/m]
f [m]
[pixel],αααα ββββNon-square pixels
f = focal length
uo, vo = offset
→ non-square pixels,αααα ββββ
Lecture 8 -Fei-Fei Li 19-Oct-117
Projective camera
=
10100
00
0
z
y
x
v
us
P' o
o
βα
f
Oc
K has 5 degrees of freedom!
Pc
P’
f = focal length
uo, vo = offset
→ non-square pixels,αααα ββββθ = skew angle
Lecture 8 -Fei-Fei Li 19-Oct-118
Projective cameraf
Oc
−
=′
1
z
y
x
0100
0v0
0ucot
P o
o
sinθθθθββββ
θθθθαααααααα
Pc
P’
f = focal length
uo, vo = offset
→ non-square pixels,αααα ββββθ = skew angle
K has 5 degrees of freedom!
Lecture 8 -Fei-Fei Li 19-Oct-119
Projective cameraf
Oc
Pc
Ow
iw
kw
jwR,T
P’
f = focal length
uo, vo = offset
→ non-square pixels,αααα ββββθ = skew angle
R,T = rotation, translation
wPTR
P44
10 ×
=
cORT~−=
Lecture 8 -Fei-Fei Li 19-Oct-1110
Projective camera
f = focal length
uo, vo = offset
→ non-square pixels,αααα ββββ
f
Oc
P
Ow
iw
kw
jw
wPMP =′
[ ] wPTRK=Internal (intrinsic) parameters
External (extrinsic) parameters
θ = skew angle
R,T = rotation, translation
P’
R,T
Lecture 8 -Fei-Fei Li 19-Oct-1111
Projective camera
wPMP =′ [ ] wPTRK=Internal (intrinsic) parameters
External (extrinsic) parameters
Lecture 8 -Fei-Fei Li 19-Oct-1112
Projective camera
wPMP =′ [ ] wPTRK=
−
=100
v0
ucot
K o
o
sinθθθθββββ
θθθθαααααααα
=T3
T2
T1
R
r
r
r
=
z
y
x
t
t
t
T
43×
Lecture 8 -Fei-Fei Li 19-Oct-1113
Goal of calibration
wPMP =′ [ ] wPTRK=
−
=100
v0
ucot
K o
o
sinθθθθββββ
θθθθαααααααα
=T3
T2
T1
R
r
r
r
=
z
y
x
t
t
t
T
43×
Estimate intrinsic and extrinsic parameters
from 1 or multiple images
Lecture 8 -Fei-Fei Li
What we will learn today?
• Review camera parameters
• Affine camera model (Problem Set 2 (Q4))
• Camera calibration
• Vanishing points and lines
19-Oct-1114
Reading:
• [FP] Chapter 3
• [HZ] Chapter 7, 8.6
Lecture 8 -Fei-Fei Li 19-Oct-1115
Weak perspective projection
Relative scene depth is small compared to its distance from the camera
= magnification
−=−=
myy
mxx
'
'
0
'where
z
fm −=
Lecture 8 -Fei-Fei Li 19-Oct-1116
Orthographic (affine) projection
Distance from center of projection to image plane is infinite
==
y'y
x'x
Lecture 8 -Fei-Fei Li 19-Oct-1117
Affine cameras
[ ] PTRKP ='
=100
00
0s
K y
x
αααααααα
=10
TR
1000
0010
0001
KM
Affine case
Parallel projection matrix
=10
TR
0100
0010
0001
KM
=100
y0
xs
K oy
ox
αααααααα
Projective caseCompared to
Lecture 8 -Fei-Fei Li 19-Oct-1118
Remember….
Projectivities:
=
=
1
y
x
H
1
y
x
bv
tA
1
'y
'x
p
Affinities:
=
=
1
y
x
H
1
y
x
10
tA
1
'y
'x
a
Lecture 8 -Fei-Fei Li 19-Oct-1119
[ ] PTRKP ='
=100
00
00
y
x
K αα
=10
TR
1000
0010
0001
KM
=
=×
×=10
bA
1000
]affine44[
1000
0010
0001
]affine33[ 2232221
1131211
baaa
baaa
M
=+=
+
=
=
1'
2
1
232221
131211 PMP
b
b
Z
Y
X
aaa
aaa
y
xP EucbA
[ ]bAMM Euc ==
We can obtain a more compact formulation than:
Affine cameras
Lecture 8 -Fei-Fei Li 19-Oct-1120
Affine cameras
PP’
P’
;1
'
=+=
=
PbAP M
v
uP [ ]bAM =
M = camera matrix
[non-homogeneous image coordinates]
To recap:
This notation is useful when we’ll discuss affine structure from motion
Lecture 8 -Fei-Fei Li 19-Oct-1121
Affine cameras
• Weak perspective much simpler math.– Accurate when object is small and distant.
– Most useful for recognition.
• Pinhole perspective much more accurate for scenes.– Used in structure from motion.
Lecture 8 -Fei-Fei Li 19-Oct-1122
The Kangxi Emperor's Southern Inspection Tour (1691-1698) By Wang HuiYou tube video – click here
Weak perspective projection - examples
Lecture 8 -Fei-Fei Li 19-Oct-1123
Weak perspective projection - examples
Qingming Festival by the Riverside Zhang Zeduan ~900 AD
Lecture 8 -Fei-Fei Li
What we will learn today?
• Review camera parameters
• Affine camera model
• Camera calibration
• Vanishing points and lines
19-Oct-1124
Reading:
• [FP] Chapter 3
• [HZ] Chapter 7, 8.6
Lecture 8 -Fei-Fei Li 19-Oct-1125
Calibration Problem
•P1… Pn with known positions in [Ow, iw, jw, kw]
•p1, … pn known positions in the image
Goal: compute intrinsic and extrinsic parameters
jC
Calibration rig
Lecture 8 -Fei-Fei Li
Remember the “digital Michelangelo project”?
19-Oct-1126
Lecture 8 -Fei-Fei Li 19-Oct-1127
Calibration Problem
jC
Calibration rig
How many correspondences do we need?
•M has 11 unknown • We need 11 equations • 6 correspondences would do it
Lecture 8 -Fei-Fei Li 19-Oct-1128
Calibration Problem
imagejC
Calibration rig
In practice: user may need to look at the
image and select the n>=6 correspondences
Lecture 8 -Fei-Fei Li 19-Oct-1129
Calibration Problem
jC
ii PMp →
=→
i
ii v
up
=
3
2
1
M
m
m
m
⋅⋅⋅⋅
=
i
i
i
i
P
PP
P
3
2
3
1
mmmm
in pixels
Lecture 8 -Fei-Fei Li 19-Oct-1130
Calibration Problem
i3
i1i P
Pu
mm=
i2i3i P)P(v mm =→
i1i3i P)P(u mm =→
i3
i2i P
Pv
mm=
i
i
v
u
=
i3
i2
i3
i1
P
PP
P
mmmm
0)( 23 =−→ iii PPv mm
0)( 13 =−→ iii PPu mm
Lecture 8 -Fei-Fei Li 19-Oct-1131
Calibration Problem
……
0)( 12131 =− PPv mm
0)( 11131 =− PPu mm
0)( 23 =− iii PPv mm
0)( 13 =− iii PPu mm
0)( 23 =− nnn PPv mm
0)( 13 =− nnn PPu mm
Lecture 8 -Fei-Fei Li 19-Oct-1132
Block Matrix Multiplication
=
=
2221
1211
2221
1211
BB
BBB
AA
AAA
What is AB ?
++++
=2222122121221121
2212121121121111
BABABABA
BABABABAAB
Lecture 8 -Fei-Fei Li 19-Oct-1133
Calibration Problem
2n x 12 12x1
1x4
=T3
T2
T1
def
m
m
m
m
4x1
…
Homogenous linear system
knownunknown
0)( 12131 =+− PPv mm
0)( 11131 =+− PPu mm
0)( 23 =+− nnn PPv mm
0)( 13 =+− nnn PPu mm
Lecture 8 -Fei-Fei Li 19-Oct-1134
Homogeneous M x N Linear Systems
A x 0=
Rectangular system (M>N)
• 0 is always a solution
Minimize |Ax|2
under the constraint |x|2 =1
M=number of equations
N=number of unknown
• To find non-zero solution
Lecture 8 -Fei-Fei Li 19-Oct-1135
How do we solve this homogenous linear system?
Calibration Problem
Singular Value Decomposition (SVD)
Lecture 8 -Fei-Fei Li 19-Oct-1136
Calibration Problem
1212T
121212n2 VDU ×××
Last column of V gives m
MiPM ip→
Compute SVD
decomposition of P
Why? See page 593 of
Hartley & Zisserman
Lecture 8 -Fei-Fei Li 19-Oct-1137
Extracting camera parameters
A
=T3
T2
T1
A
a
a
a
[ ]TRK=
3
1
a±=ρρρρ
=
3
2
1
b
b
b
b
Estimated values
)(u 212
o aa ⋅= ρρρρ)(v 32
2o aa ⋅= ρρρρ
( ) ( )3231
3231cosaaaaaaaa
×⋅××⋅×=θθθθ
Intrinsic
b
−
=100
v0
ucot
K o
o
sinθθθθββββ
θθθθααααααααρ
Lecture 8 -Fei-Fei Li 19-Oct-1138
Theorem (Faugeras, 1993)
[ ] [ ] ][ bATKRKTRKM ===
=
3
2
1
a
a
a
A
=100
0 y
x
c
cs
K βα
lf
;kf
==
ββββαααα
Lecture 8 -Fei-Fei Li 19-Oct-1139
Extracting camera parameters
A
=T3
T2
T1
A
a
a
a
[ ]TRK=
=
3
2
1
b
b
b
b
Estimated values
Intrinsic
θθθθρρρραααα sin312 aa ×=
θθθθρρρρββββ sin322 aa ×=
b
f
ρ
Lecture 8 -Fei-Fei Li 19-Oct-1140
Extracting camera parameters
Extrinsic
( )32
321 aa
aar
××=
33
1
ar
±=
132 rrr ×= b1KT −= ρρρρ
A
=T3
T2
T1
A
a
a
a
[ ]TRK=
=
3
2
1
b
b
b
b
Estimated values
b
ρ
Lecture 8 -Fei-Fei Li 19-Oct-1141
Calibration DemoCamera Calibration Toolbox for Matlab J. Bouguet – [1998-2000]
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples
Lecture 8 -Fei-Fei Li 19-Oct-1142
Calibration Demo
Lecture 8 -Fei-Fei Li 19-Oct-1143
Calibration Demo
Lecture 8 -Fei-Fei Li 19-Oct-1144
Calibration Demo
Lecture 8 -Fei-Fei Li 19-Oct-1145
Calibration Demo
Lecture 8 -Fei-Fei Li 19-Oct-1146
Calibration Demo
Lecture 8 -Fei-Fei Li 19-Oct-1147
Calibration Demo
Lecture 8 -Fei-Fei Li 19-Oct-1148
Calibration Demo
Lecture 8 -Fei-Fei Li
What we will learn today?
• Review camera parameters
• Affine camera model
• Camera calibration
• Vanishing points and lines (Problem Set 2
(Q1))
19-Oct-1149
Reading:
• [FP] Chapter 3
• [HZ] Chapter 7, 8.6
Lecture 8 -Fei-Fei Li 19-Oct-1150
Properties of Projection
•Points project to points
•Lines project to lines
Lecture 8 -Fei-Fei Li 19-Oct-1151
Properties of Projection
Vanishing point•Angles are not preserved
•Parallel lines meet
Lecture 8 -Fei-Fei Li 19-Oct-1152
Lines in a 2D plane
0cbyax =++
-c/b
-a/b
=c
b
a
l
If x = [ x1, x2]T ∈ l 0
c
b
a
1
x
xT
2
1
=
l
x
y
Lecture 8 -Fei-Fei Li 19-Oct-1153
Lines in a 2D plane
Intersecting lines
llx ′×= l
l′
Proof
lll ⊥′×lll ′⊥′×
0l)ll( =⋅′×→0l)ll( =′⋅′×→
lx∈→
x
lx ′∈→
→ x is the intersecting point
x
y
x
Lecture 8 -Fei-Fei Li 19-Oct-1154
Points at infinity (ideal points)
0x,
x
x
x
x 3
3
2
1
≠
=
=c
b
a
l
′=′
c
b
a
l
−′−=′×→0
a
b
)cc(llLet’s intersect two parallel lines:
Agree with the general idea of two lines intersecting at infinity
l
l′
=∞
02
1
x
x
x
Lecture 8 -Fei-Fei Li 19-Oct-1155
Lines at infinity ∞l
Set of ideal points lies on a line called the line at infinity
How does it look like?
∞l
=∞
1
0
0
l
T
2
1
0
x
x
0
1
0
0
=
Indeed:
Lecture 8 -Fei-Fei Li 19-Oct-1156
Projective projections of lines at infinity (2D)
lHl T−=′
=
bv
tAH
?lH T =∞−
=
=
−
b
t
t
1
0
0
bv
tAy
xT
is it a line at infinity?
…no!
?lH TA =∞−
=
−=
=
−
−−
1
0
0
1
0
0
1
0
1
0
0
10 TT
TT
At
AtA
Lecture 8 -Fei-Fei Li 19-Oct-1157
horizon
- Recognize the horizon line
- Measure if the 2 lines meet
at the horizon
- if yes, these 2 lines are //
•Recognition helps reconstruction!
•Humans have learnt this
Are these two lines parallel or not?
∞−= lHl T
hor
Projective projections of lines at infinity (2D)
Lecture 8 -Fei-Fei Li 19-Oct-1158
Vanishing points (= ideal points in 2D)
[ ]TRKM =
dv K=
d=direction of the line
d
C
vVanishing points
= points where
parallel lines
intersect in 3D
Image of a vanishing point =
Lecture 8 -Fei-Fei Li 19-Oct-1159
Horizon
•Sets of parallel lines on the same plane lead to collinear vanishing
points [The line is called the horizon for that plane]
horizon
Lecture 8 -Fei-Fei Li 19-Oct-1160
C
n
horizTK ln =
horizl
Horizon
Lecture 8 -Fei-Fei Li 19-Oct-1161
Criminisi & Zisserman, 99
ApplicationThese transformations are used in single view metrology
Lecture 8 -Fei-Fei Li 19-Oct-1162
Applicationthese transformations are used in single view metrology
Criminisi & Zisserman, 99
Lecture 8 -Fei-Fei Li 19-Oct-1163
La Trinita' (1426)
Firenze, Santa Maria
Novella; by Masaccio
(1401-1428)
Criminisi & Zisserman, 99
Applicationthese transformations are used in single view metrology
Lecture 8 -Fei-Fei Li 19-Oct-1164
Lecture 8 -Fei-Fei Li 19-Oct-1165
Hoiem et al, 05…
Applicationthese transformations are used in single view metrology
http://www.cs.uiuc.edu/homes/dhoiem/projects/software.html
Lecture 8 -Fei-Fei Li
A software: Make3D“Convert your image into 3d model”
Saxena, Sun, Ng, 05…
http://make3d.stanford.edu/
http://make3d.stanford.edu/images/view3D/185
http://make3d.stanford.edu/images/view3D/931?noforward=true
http://make3d.stanford.edu/images/view3D/108
Applicationthese transformations are used in single view metrology
19-Oct-1166
Lecture 8 -Fei-Fei Li
What we have learned today
• Review camera parameters
• Affine camera model (Problem Set 2 (Q4))
• Camera calibration
• Vanishing points and lines (Problem Set 2
(Q1))
19-Oct-1167
Reading:
• [FP] Chapter 3
• [HZ] Chapter 7, 8.6
Lecture 8 -Fei-Fei Li
Supplementary Materials
19-Oct-1168
Lecture 8 -Fei-Fei Li
Degeneracy and distortion in
real-world camera calibration
19-Oct-1169
Lecture 8 -Fei-Fei Li 19-Oct-1170
Degenerate cases
•Pi’s cannot lie on the same plane!
• Points cannot lie on the intersection curve of two
quadric surfaces
Lecture 8 -Fei-Fei Li 19-Oct-1171
Radial Distortion
No distortion
Pin cushion
Barrel
– Caused by imperfect lenses
– Deviations are most noticeable for rays that pass through the
edge of the lens
Lecture 8 -Fei-Fei Li 19-Oct-1172
Radial Distortion
ii
ii p
v
uPM
100
00
00
1
1
=
→
λλλλ
λλλλ
d
v
vucvbuad 222 ++=
u
∑±==
3
1p
2ppdκ1λ
Polynomial function
Distortion coefficient
To model radial behavior
Lecture 8 -Fei-Fei Li 19-Oct-1173
Radial Distortion
=
i
ii v
up
=
3
2
1
Q
q
q
q
=
i3
i2
i3
i1
P
PP
P
qqqq
ii
ii p
v
uPM
100
00
00
1
1
=
→
λλλλ
λλλλ
Q
==
PPv
PPu
2i3i
i1i3i
Non-linear system of equations
Lecture 8 -Fei-Fei Li 19-Oct-1174
General Calibration Problem
)(PfX =
measurement parameter
f( ) is nonlinear
-Newton Method
-Levenberg-Marquardt Algorithm
• Iterative, starts from initial solution
• May be slow if initial solution far from real solution
• Estimated solution may be function of the initial solution
• Newton requires the computation of J, H
• Levenberg-Marquardt doesn’t require the computation of H
Lecture 8 -Fei-Fei Li 19-Oct-1175
General Calibration Problem
A possible algorithm
1. Solve linear part of the system to find approximated solution
2. Use this solution as initial condition for the full system
3. Solve full system (including distortion) using Newton or L.M.
)(PfX =
measurement parameter
f( ) is nonlinear
Lecture 8 -Fei-Fei Li 19-Oct-1176
General Calibration Problem
)(PfX =
measurement parameter
f( ) is nonlinear
Typical assumptions for computing initial condition :
- zero-skew, square pixel
- uo, vo = known center of the image
- no distortion
Just estimate f
and R, T
Lecture 8 -Fei-Fei Li 19-Oct-1177
Tsai’s calibration technique
1. Estimate m1 and m2 first:
=
i
ii v
up
=
i3
i2
i3
i1
P
PP
P
1
mmmm
λλλλHow to do that?
d
v
uHint: slopev
u
i
i =
Lecture 8 -Fei-Fei Li 19-Oct-1178
Tsai’s calibration technique
1. Estimate m1 and m2 first:
=
i
ii v
up
=
i3
i2
i3
i1
P
PP
P
1
mmmm
λλλλ
0)()( 121111 =− PuPv mm
0)()( 21 =− iiii PuPv mm
0)()( 21 =− nnnn PuPv mm
…
0Q =n
=
2
1
m
mn
i
i
i
i
i
i
i
i
P
P
P
P
P
P
v
u
2
1
3
2
3
1
)()(
)()(
mm
mmmm
==
Lecture 8 -Fei-Fei Li 19-Oct-1179
Tsai’s calibration technique
2. Once that m1 and m2 are estimated, estimate m3:
=
i
ii v
up
=
i3
i2
i3
i1
P
PP
P
1
mmmm
λλλλ
3m is non linear function of 1m 2m λλλλ
There are some degenerate configurations for which m1 and m2 cannot be computed