PhotoGRammetry · photogrammetry: aerial, terrestrial, and close ranger. ... what is a stereoplotter?
Direct Linear Transformation & Computer Vision Models · CE 59700: Digital Photogrammetric Systems...
Transcript of Direct Linear Transformation & Computer Vision Models · CE 59700: Digital Photogrammetric Systems...
CE 59700: Digital Photogrammetric Systems Ayman F. Habib1
Direct Linear Transformation & Computer Vision Models
Chapter 7-A4
CE 59700: Digital Photogrammetric Systems Ayman F. Habib2
Photogrammetry Vs. Computer Vision• Conventional Photogrammetry is focusing on precise
geometric information extraction from imagery.– Topographic mapping from space borne and airborne imagery– Metrological information extraction through close-range
photogrammetry (terrestrial photogrammetry)• Object-to-camera distance is less than 100meter
• Computer Vision (CV) is mainly concerned with automated image understanding:– Object recognition,– Navigation and obstacle avoidance, and– Object modeling
CE 59700: Digital Photogrammetric Systems Ayman F. Habib3
Airborne Photogrammetric Mapping
CE 59700: Digital Photogrammetric Systems Ayman F. Habib4
Airborne Photogrammetric Mapping
CE 59700: Digital Photogrammetric Systems Ayman F. Habib5
Close-Range Photogrammetric Mapping
CE 59700: Digital Photogrammetric Systems Ayman F. Habib6
CV: Object Recognition
CE 59700: Digital Photogrammetric Systems Ayman F. Habib7
CV: Navigation & Obstacle Avoidance
CE 59700: Digital Photogrammetric Systems Ayman F. Habib8
Photogrammetry Vs. Computer Vision• Photogrammetry is always concerned with precise
geometric information extraction.– Photogrammetric mapping considers potential deviations from
the assumed perspective projection.
• For Computer Vision (CV):– Focus is always on automation.– Object recognition and navigation applications do not require
precise derivation of geometric information.– Depending on the application, object modeling might require
precise geometric information extraction.– CV usually assumes that the collinearity of the object point,
perspective center, and corresponding image point is maintained, even for un-calibrated cameras.
CE 59700: Digital Photogrammetric Systems Ayman F. Habib9
Object-to-Image Coordinate Transformation in Photogrammetry
Collinearity Equations
CE 59700: Digital Photogrammetric Systems Ayman F. Habib10
o
a
A
oa = oA
These vectors should be defined w.r.t.the same coordinate system.
Collinearity Equations
CE 59700: Digital Photogrammetric Systems Ayman F. Habib11
Oi
xc
yczc
A
XA
YA
ZA
a+
(xa, ya)
R( , , )
XG
YG
ZG
OG
pp+
c
(Perspective Center)
XO
ZO
YO
Collinearity Equations
CE 59700: Digital Photogrammetric Systems Ayman F. Habib12
Collinearity Equations
The vector connecting the perspective center to the image point
w.r.t. the image coordinate system
o
a
c
distyydistxx
cyx
distydistx
rv ypa
xpa
p
p
ya
xac
oai
0
CE 59700: Digital Photogrammetric Systems Ayman F. Habib13
Collinearity Equations
The vector connecting the perspective center to the object point
w.r.t. the ground coordinate system
o
A
oA
oA
oA
o
o
o
A
A
Am
oAo
ZZYYXX
ZYX
ZYX
rV
CE 59700: Digital Photogrammetric Systems Ayman F. Habib14
Where: is a scale factor (+ve).
Collinearity Equations
11 12 13
21 22 23
31 32 33
( , , )c c mi oa O m oA
a p x A o
a p y A o
A o
v r M V R rx x dist m m m X Xy y dist m m m Y Y
c m m m Z Z
oAoa
CE 59700: Digital Photogrammetric Systems Ayman F. Habib15
Collinearity EquationscmRM
yoAoAoA
oAoAoApa
xoAoAoA
oAoAoApa
distZZmYYmXXmZZmYYmXXmcyy
distZZmYYmXXmZZmYYmXXmcxx
)()()()()()()()()()()()(
333231
232221
333231
131211
mcRR
yoAoAoA
oAoAoApa
xoAoAoA
oAoAoApa
distZZrYYrXXrZZrYYrXXrcyy
distZZrYYrXXrZZrYYrXXrcxx
)()()()()()()()()()()()(
332313
322212
332313
312111
CE 59700: Digital Photogrammetric Systems Ayman F. Habib16
Object-to-Image Coordinate Transformation
Direct Linear TransformationComputer Vision Model
CE 59700: Digital Photogrammetric Systems Ayman F. Habib17
DLT & Computer Vision Models• The DLT and computer vision models encompass:
– Collinearity Equations,– Non-orthogonality () between the axes of the image/camera
coordinate system, and– Two scale factors (Sx, Sy) along the axes of the image
coordinate system.
• DLT & CV models can directly deal with pixel coordinates.
• We will start with modifying the rotation matrix to consider the impact of the non-orthogonality ().– Primary rotation @ the -axis of the ground coord. system– Secondary rotation @ the -axis– Tertiary rotation & @ the -axis
CE 59700: Digital Photogrammetric Systems Ayman F. Habib18
Primary Rotation ()
X
ZY
& X
YZ
CE 59700: Digital Photogrammetric Systems Ayman F. Habib19
Primary Rotation ()
zyx
Rzyx
zyx
zyx
cossin0sincos0001
CE 59700: Digital Photogrammetric Systems Ayman F. Habib20
Secondary Rotation ()
Y
Z
X
X
Z
& Y
CE 59700: Digital Photogrammetric Systems Ayman F. Habib21
Secondary Rotation ()
zyx
Rzyx
zyx
zyx
cos0sin010
sin0cos
CE 59700: Digital Photogrammetric Systems Ayman F. Habib22
Tertiary Rotation ()
X
Z
YY
X
& Z
CE 59700: Digital Photogrammetric Systems Ayman F. Habib23
Tertiary Rotation ()
zyx
Rzyx
zyx
zyx
1000cossin0sincos
CE 59700: Digital Photogrammetric Systems Ayman F. Habib24
Rotation in Space
zyx
RRRzyx
// to the ground coordinate system // to the image coordinate system
CE 59700: Digital Photogrammetric Systems Ayman F. Habib25
Rotation in Space
coscossinsincoscossin
cossincossinsincossin
sinsinsincoscoscossinsinsincos
sinsincos
coscos:
33
32
31
23
22
21
13
12
11
333231
232221
131211
rrrrrrrrrwhere
rrrrrrrrr
RRRR
CE 59700: Digital Photogrammetric Systems Ayman F. Habib26
X
Z
YY
X
& Z
Consideration of the Non-Orthogonality ()
ZYX
ZYX
1000)cos(sin0)sin(cos
CE 59700: Digital Photogrammetric Systems Ayman F. Habib27
ZYX
ZYX
1000)cos(sin0)sin(cos
ZYX
ZYX
1000sincossin0cossincos
Consideration of the Non-Orthogonality ()
sincossinsincoscos)cos(cossinsincoscossin)sin(
Assuming small non-orthogonality angle ()
CE 59700: Digital Photogrammetric Systems Ayman F. Habib28
Consideration of the Non-Orthogonality ()
10001001
1000cossin0sincos
1000sincossin0cossincos
ZYX
RZYX
RRRZYX
10001001
10001001
CE 59700: Digital Photogrammetric Systems Ayman F. Habib29
Consideration of the Non-Orthogonality ()
ZYX
RZYX
zyx
T
10001001
// to the image coordinate system // to the ground coordinate system
ZYX
RZYX
10001001
Note: 1 00 1 00 0 1
1 00 1 00 0 1
CE 59700: Digital Photogrammetric Systems Ayman F. Habib30
Consideration of the Non-Orthogonality ()• Collinearity Equations while considering the non-
orthogonality () between the axes of the image coordinate system.
O
O
OT
p
p
ZZYYXX
Rcyyxx
10001001
CE 59700: Digital Photogrammetric Systems Ayman F. Habib31
O
O
OT
yp
xp
ZZYYXX
Rc
syysxx
10001001
/)(/)(
O
O
OT
yp
xp
ZZYYXX
Rccsyycsxx
10001001
/1
)/()()/()(
• Divide both sides by (-c).
Consideration of the Scale Factors• Collinearity Equations while considering the non-
orthogonality () between the axes of the image coordinate system & different scale factors.
CE 59700: Digital Photogrammetric Systems Ayman F. Habib32
Consideration of the Scale Factors
O
O
OT
yp
xp
ZZYYXX
Rcyycxx
10001001
1)/()()/()(
O
O
OT
p
p
y
x
ZZYYXX
Ryyxx
cc
10001001
1)()(
1000/1000/1
O
O
OT
y
x
p
p
ZZYYXX
Rcc
yyxx
10001001
1000000
1)()(
• csx→ cx , csy→ cy & -/c → `.
CE 59700: Digital Photogrammetric Systems Ayman F. Habib33
DLT & Computer Vision Models
O
O
OT
y
xx
p
p
ZZYYXX
Rccc
yyxx
100000
1)()(
O
O
O
y
xx
p
p
ZZYYXX
rrrrrrrrr
ccc
yyxx
332313
322212
312111
100000
1)()(
&
CE 59700: Digital Photogrammetric Systems Ayman F. Habib34
DLT & Computer Vision Models
′
′
′
′
CE 59700: Digital Photogrammetric Systems Ayman F. Habib35
DLT & Computer Vision Models
′
Where:
CE 59700: Digital Photogrammetric Systems Ayman F. Habib36
DLT & Computer Vision Models
3
3
'1
1
0 { }0 0 1
{ }
TO
x x p
y p
TO
Xx
Yy K R I X
Z
c c xK c y Calibration Matrix
R I X Exterior Orientation Matrix
CE 59700: Digital Photogrammetric Systems Ayman F. Habib37
DLT & Computer Vision Models• The Direct Linear Transformation (DLT), which has
been developed by the photogrammetric community, is an alternative to the collinearity equations that allows for direct transformation between machine/pixel coordinates and corresponding ground coordinates.– &
• The DLT can be also represented by the following form:
–1 1
CE 59700: Digital Photogrammetric Systems Ayman F. Habib38
DLT & Computer Vision Models
1 2 3
5 6 7
9 10 11
' 00 0 1
x x pT
y p
L L L c c xD L L L c y R
L L L
4
8
12
' 00 0 1
x x p OT
y p O
O
L c c x XL c y R YL Z
DLT: Direct Linear Transformation
CE 59700: Digital Photogrammetric Systems Ayman F. Habib39
DLT & CV Models: Pixel Coordinates• The DLT & CV models can also consider the direct
transformation from pixel to ground coordinates.
⁄ _ _⁄ _ _
u
v
CE 59700: Digital Photogrammetric Systems Ayman F. Habib40
DLT & CV Models: Pixel Coordinates⁄ _ _
⁄ _ _
_ _ ⁄ _ __ _ ⁄ _ _
_ _ ⁄ _ __ _ ⁄ _ _
CE 59700: Digital Photogrammetric Systems Ayman F. Habib41
DLT & CV Models: Pixel Coordinates_ _ ⁄ _ _
_ _ ⁄ _ _
_ _ ⁄ _ __ _ ⁄ _ _
_ _ ⁄ _ __ _ ⁄ _ _
_ _⁄ ⁄_ _⁄ ⁄
_ _⁄ ⁄_ _⁄ ⁄
CE 59700: Digital Photogrammetric Systems Ayman F. Habib42
DLT & CV Models: Pixel Coordinates• Modified Calibration Matrix:
_ _⁄ ⁄_ _⁄ ⁄
_ _⁄ _ _⁄ _ _⁄ ⁄0 _ _⁄ _ _⁄ ⁄0 0 1
CE 59700: Digital Photogrammetric Systems Ayman F. Habib43
DLT & CV Models: Pixel Coordinates• For DLT when working with pixel coordinates, we have
the following model.
–
•
•
CE 59700: Digital Photogrammetric Systems Ayman F. Habib44
Modern Photogrammetry & Computer Vision• Modern Photogrammetry and Computer Vision are
converging fields.
Art and science of tool development for automatic generation of spatial and descriptive information from
multi-sensory data and/or systems
CE 59700: Digital Photogrammetric Systems Ayman F. Habib45
DLT → IOPs & EOPs
Approach # 1
CE 59700: Digital Photogrammetric Systems Ayman F. Habib46
DLT → IOP & EOP
1 2 3
5 6 7
9 10 11
00 0 1
x x pT
y p
L L L c c xD L L L c y R
L L L
4
8
12
00 0 1
x x p OT
y p O
O
L c c x XL c y R YL Z
CE 59700: Digital Photogrammetric Systems Ayman F. Habib47
DLT → IOP & EOP
No Sign Ambiguity
O
O
O
ZYX
LLLLLLLLL
LLL
11109
765
321
12
8
4
• Given:
• Then:
12
8
41
11109
765
321
LLL
LLLLLLLLL
ZYX
O
O
O
CE 59700: Digital Photogrammetric Systems Ayman F. Habib48
DLT → IOP & EOP
1000
10002
pp
yx
x
py
pxxT
yxcc
cycxcc
DD
1173
1062
951
11109
765
3212)()(
LLLLLLLLL
LLLLLLLLL
KKRKRKDD TTTTT
2211
210
2933)( LLLDD T
}{211
210
29 AmbiguitySignLLL
Then:
CE 59700: Digital Photogrammetric Systems Ayman F. Habib49
DLT → IOP & EOP
pT xLLLLLLDD 2
3112101913)(
)()(
211
210
29
31121019LLL
LLLLLLxp
No Sign Ambiguity
Then:
1173
1062
951
11109
765
3212)()(
LLLLLLLLL
LLLLLLLLL
KKRKRKDD TTTTT
CE 59700: Digital Photogrammetric Systems Ayman F. Habib50
pT yLLLLLLDD 2
7116105923)(
DLT → IOP & EOP
)()(
211
210
29
71161059LLL
LLLLLLyp
No Sign Ambiguity
Then:
1173
1062
951
11109
765
3212)()(
LLLLLLLLL
LLLLLLLLL
KKRKRKDD TTTTT
CE 59700: Digital Photogrammetric Systems Ayman F. Habib51
)()( 22227
26
2522 yp
T cyLLLDD
DLT → IOP & EOP
1173
1062
951
11109
765
3212)()(
LLLLLLLLL
LLLLLLLLL
KKRKRKDD TTTTT
5.02
211
210
29
27
26
25
)(
py yLLL
LLLc
No Sign Ambiguity
Then:
CE 59700: Digital Photogrammetric Systems Ayman F. Habib52
DLT → IOP & EOP
)()( 273625121 ppyx
T yxccLLLLLLDD
1173
1062
951
11109
765
3212)()(
LLLLLLLLL
LLLLLLLLL
KKRKRKDD TTTTT
ppyx yxLLL
LLLLLLcc )(/1 211
210
29
736251
No Sign Ambiguity
Then:
CE 59700: Digital Photogrammetric Systems Ayman F. Habib53
)()( 2222223
22
2111 pxx
T xccLLLDD
DLT → IOP & EOP
1173
1062
951
11109
765
3212)()(
LLLLLLLLL
LLLLLLLLL
KKRKRKDD TTTTT
5.0222
211
210
29
23
22
21
)(
pxx xcLLL
LLLc
No Sign Ambiguity
Then:
CE 59700: Digital Photogrammetric Systems Ayman F. Habib54
DLT → IOP & EOP• Given:
• Then: sin139 rL
Sign Ambiguity
332313
322212
312111
11109
765
321
1000
rrrrrrrrr
ycxcc
LLLLLLLLL
py
pxx
CE 59700: Digital Photogrammetric Systems Ayman F. Habib55
Collinearity Equations• Objective: Resolve the sign ambiguity in
• Since the scale factor is always +ve
• Assuming that the origin (0, 0, 0) is visible in the imagery
veZZrYYrXXr OOO )()()( 332313
veZrYrXr OOO 332313
)()()()()()()()()(
332313
322212
312111
OOO
OOO
OOO
p
p
ZZrYYrXXrZZrYYrXXrZZrYYrXXr
Scyyxx
CE 59700: Digital Photogrammetric Systems Ayman F. Habib56
DLT → IOP & EOP
• By choosing L12 = 1.
12 13 23 33
13 23 33
13 23 33
( )1 ( )
1( )
O O O
O O O
O O O
L r X r Y r Zr X r Y r Z
r X r Y r Z
is Negative
211
210
29 LLL
O
O
OT
py
pxx
ZYX
Rycxcc
LLL
1000
12
8
4
CE 59700: Digital Photogrammetric Systems Ayman F. Habib57
DLT → IOP & EOP
• No sign Ambiguity
211
210
29
9
139
sin
sin
LLLLrL
CE 59700: Digital Photogrammetric Systems Ayman F. Habib58
DLT → IOP & EOP
coscoscossin
3311
2310
rLrL
11
10tan LL
No Sign Ambiguity
332313
322212
312111
11109
765
321
1000
rrrrrrrrr
ycxcc
LLLLLLLLL
py
pxx
CE 59700: Digital Photogrammetric Systems Ayman F. Habib59
DLT → IOP & EOP
1000
1000
1
11109
765
321
333231
232221
131211
332313
322212
312111
11109
765
321
py
pxx
py
pxx
ycxcc
LLLLLLLLL
rrrrrrrrr
rrrrrrrrr
ycxcc
LLLLLLLLL
• Retrieve
• Note: There is an ambiguity in determination ( cannot be distinguished).
coscos 11r
CE 59700: Digital Photogrammetric Systems Ayman F. Habib60
DLT → IOP & EOP
Approach # 2: Matrix Factorization
CE 59700: Digital Photogrammetric Systems Ayman F. Habib61
DLT → IOP (Factorization # 1)• Conceptual basis: Direct derivation of the calibration matrix
• Cholesky Decomposition of DDT→ K (Calibration Matrix)? Wrong
1000
1000
)()(
2
1173
1062
951
11109
765
3212
pp
yx
x
py
pxxT
TTTTT
yxcc
cycxcc
DD
LLLLLLLLL
LLLLLLLLL
KKRKRKDD
CE 59700: Digital Photogrammetric Systems Ayman F. Habib62
1
1)(
NMMMNCHO
T TMM
TDDN TKK
TT
T
KKMMN
MMN21
1
1
11)]}({[ TDDCHOK
TKK
DLT → IOP (Factorization # 2)
CE 59700: Digital Photogrammetric Systems Ayman F. Habib63
1000
1000
1
11109
765
321
333231
232221
131211
332313
322212
312111
11109
765
321
py
pxx
py
pxx
ycxcc
LLLLLLLLL
rrrrrrrrr
rrrrrrrrr
ycxcc
LLLLLLLLL
• Using the rotation matrix R, one can derive the individual rotation angles , and .
DLT → Rotation Angles
CE 59700: Digital Photogrammetric Systems Ayman F. Habib64
Analysis
CE 59700: Digital Photogrammetric Systems Ayman F. Habib65
Perspective Center
1211109
8765
4321
12109
765
321
12
8
4
LZLYLXLLZLYLXLLZLYLXL
ZYX
LLLLLLLLL
LLL
OOO
OOO
OOO
O
O
O
• (XO, YO , ZO) is the intersection point of three different planes whose surface normals are (L1, L2, L3), (L5, L6, L7) and (L9, L10, L11), respectively.
CE 59700: Digital Photogrammetric Systems Ayman F. Habib66
Perspective Center
332313
322212
312111
11109
765
321
1000
rrrrrrrrr
ycxcc
LLLLLLLLL
py
pxx
• Assuming:– xp ≈ 0.0 and yp ≈ 0.0– -cx ≈ 0.0
332313
322212
312111
11109
765
321
rrrrcrcrcrcrcrc
LLLLLLLLL
yyy
xxx
• The three surfaces are orthogonal to each other.– This would lead to better intersection.
CE 59700: Digital Photogrammetric Systems Ayman F. Habib67
332313
322212
312111
11109
765
321
1000
rrrrrrrrr
ycxcc
LLLLLLLLL
py
pxx
• Assuming:– xp ≠ 0.0 and yp ≠ 0.0– -cx ≈ 0.0
332313
333223221312
333123211311
11109
765
321
rrrryrcryrcryrcrxrcrxrcrxrc
LLLLLLLLL
pypypy
pxpxpx
• As xp and yp increase, the surface normals become almost parallel.– This would lead to weak intersection.
Perspective Center
CE 59700: Digital Photogrammetric Systems Ayman F. Habib68
• The rows of D are not correlated:– They are orthogonal to each other.
• L-1 is well defined.
1000
1
11109
765
321
333231
232221
131211
py
pxx
ycxcc
LLLLLLLLL
rrrrrrrrr
• Assuming:– xp ≈ 0.0 and yp ≈ 0.0– -cx ≈ 0.0
332313
322212
312111
11109
765
321
rrrrcrcrcrcrcrc
LLLLLLLLL
yyy
xxx
Rotation Angles
CE 59700: Digital Photogrammetric Systems Ayman F. Habib69
1000
1
11109
765
321
333231
232221
131211
py
pxx
ycxcc
LLLLLLLLL
rrrrrrrrr
• Assuming:– xp ≠ 0.0 and yp ≠ 0.0– -cx ≈ 0.0
332313
333223221312
333123211311
11109
765
321
rrrryrcryrcryrcrxrcrxrcrxrc
LLLLLLLLL
pypypy
pxpxpx
• The rows of D tend to be highly correlated.• L-1 is not well defined.
Rotation Angles