3D Motion Estimation. 3D model construction Video Manipulation.
-
date post
22-Dec-2015 -
Category
Documents
-
view
224 -
download
1
Transcript of 3D Motion Estimation. 3D model construction Video Manipulation.
3D Motion Estimation
3D model construction
3D model construction
Video Manipulation
Visual Motion
• Allows us to compute useful properties of the 3D world, with very little knowledge.
• Example: Time to collision
Time to Collision
ff
LL
vv
LL
DDoo
l(t)l(t)
An object of heightAn object of height L L moves with moves with
constant velocity constant velocity v:v:•At time At time t=0t=0 the object is at: the object is at:
• D(0) = DD(0) = Doo
•At time At time tt it is at it is at
•D(t) = DD(t) = Doo – vt – vt
•It will crash with the camera at It will crash with the camera at
time:time:
• D(D() = D) = Doo – v – v = 0 = 0
• = D= Doo/v/v
t=0t=0tt
D(t)D(t)
Time to Collision
ff
LL
vv
LL
DDoo
l(t)l(t)
t=0t=0tt
D(t)D(t)
The image of the object has size The image of the object has size
l(t):l(t):
Taking derivative wrt time:Taking derivative wrt time:
Time to Collision
ff
LL
vv
LL
DDoo
l(t)l(t)
t=0t=0tt
D(t)D(t)
And their ratio is:And their ratio is:
Time to Collision
ff
LL
vv
LL
DDoo
l(t)l(t)
t=0t=0tt
D(t)D(t)
And And time to collisiontime to collision::
Can be Can be directly directly measured measured from from imageimage
Can be found, without knowing Can be found, without knowing LL or or DDoo or or vv !! !!
Structure from Motion
rtzu ˆ
1tr Z
rrzu ωˆ1
rot Ft,
r
Zrottr uuu
u
Passive Navigation and Structure from Motion
. velocity rotational and ,
velocity ionalh translatmotion wit rigid a with moves system The
,T
zyxT
zyx ωωωttt t
fyxZYX T ,, points image ontoproject ,, points Scene rR
image in the observed ispoint scene a of ,, velocity 3D theand zyx VVVR
.0., velocity as vur
Consider a 3D point P and its image:
ff
PP
pp
ZZ zz
Using pinhole camera equation:Using pinhole camera equation:
Let things move:
ff
PPVV
ppvv
ZZ zz
The The relative velocityrelative velocity of P wrt camera: of P wrt camera:
TranslatioTranslation velocityn velocity Rotation Rotation
angular angular velocityvelocity
3D Relative Velocity:
ff
PPVV
ppvv
ZZ zz
The relative velocity of P wrt camera:The relative velocity of P wrt camera:
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
Taking derivative wrt time:Taking derivative wrt time:
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
TranslationaTranslational componentl component
Scaling ambiguity(t and Z can onlybe derived up to a scale Factor)
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
Rotational Rotational componentcomponent
NOTE: The rotational component is independent of depth Z !NOTE: The rotational component is independent of depth Z !
Translational flow field
(FOC).n contractio of focusor
(FOE)expansion of focus theis ,, p where 00 o
f
t
tf
t
tyx
z
y
z
x
Rotational flow field
(AOR). plane image thepierces axisrotation theepoint wher theis ,
ff
z
y
z
x
Pure Translation
What if tWhat if tzz 0 ? 0 ?
All motion field vectors are parallel to each other and All motion field vectors are parallel to each other and inversely proportional to depth !inversely proportional to depth !
Pure Translation: Properties of the MF
• If tz 0 the MF is RADIAL with all vectors pointing towards (or away from) a single point po. If tz = 0 the MF is PARALLEL.
• The length of the MF vectors is inversely proportional to depth Z. If tz 0 it is also directly proportional to the distance between p and po.
Pure Translation: Properties of the MF
• po is the vanishing point of the direction of translation.
• po is the intersection of the ray parallel to the translation vector and the image plane.
Special Case: Moving Plane
ff
PPVV
ppvv
ZZ zz
nnPlanar surfaces are Planar surfaces are common in man-made common in man-made environmentsenvironments
Question: How does the MF of a moving plane look like?Question: How does the MF of a moving plane look like?
Special Case: Moving PlanePoints on the plane must satisfy Points on the plane must satisfy the equation describing the plane. the equation describing the plane.
Let Let •nn be the unit vector normal to the be the unit vector normal to the plane.plane.•dd be the distance from the plane to be the distance from the plane to the origin.the origin.•NOTE:NOTE: If the plane is moving wrt If the plane is moving wrt camera, camera, nn and and dd are functions of timeare functions of time..
Then:Then:
dd PP
ZZ
nn
OOYY
XX
wherewhere
Special Case: Moving Plane
dd PP
ZZ
nn
OOYY
XX
Using the pinhole Using the pinhole projection projection equation:equation:pp
LetLet be the image of Pbe the image of P
Using the plane equation:Using the plane equation:
Solving for Z:Solving for Z:
Special Case: Moving Plane
And Plug in:And Plug in:
dd PP
ZZ
nn
OOYY
XX
pp
Now consider the MF equations:Now consider the MF equations:
Special Case: Moving Plane
dd PP
ZZ
nn
OOYY
XX
pp
The MF equations become:The MF equations become:
wherewhere
Special Case: Moving Plane
dd PP
ZZ
nn
OOYY
XX
pp
MF equations:MF equations:
Q: What is the significance of this?Q: What is the significance of this?
A: The MF vectors are given by low order (second) A: The MF vectors are given by low order (second) polynomials. polynomials.
•Their coeffs. aTheir coeffs. a11 to a to a8 8 (only 8 !) are functions of n, d, t and (only 8 !) are functions of n, d, t and ..•The same coeffs. can be obtained with a different plane The same coeffs. can be obtained with a different plane and relative velocity.and relative velocity.
Moving Plane: Properties of the MF
• The MF of a planar surface is at any time a quadratic function in the image coordinates.
• A plane nTP=d moving with velocity V=-t-£ P has the same MF than a plane with normal n’=t/|t| , distance d and moving with velocity
V=|t|n –( + n£ t/d)£ P
Classical Structure from Motion
• Established approach is the epipolar minimization: The “derotated flow” should be parallel to the translational flow.
uu
-urot
-urot
EE
utr
t
utr
The Translational Case(a least squares formulation)
Substitute back
min
Substitute
)),(( ||),(||
min
22
2
22
Zv
Zu
yttxtt
dydxyxfyxf
dydxZ
tytv
Z
txtu
zyzx
yzxz
min22
2
dydxvu
)minimized. be would
for which oflength the(Find . respect to with Minimize :1 Step tr dZ u
vu
ZZZ
vZZ
u
22
220
2 minimize We d
Step 2: Differentiate with respect to tx, ty, tz, set expression to zero.
222
Let
vuvuK
0III
0II
0I
dydxKxtyt
dydxKxtt
dydxKytt
yx
zx
zy
0IIIIII equationsdependent linearly 3 zyx ttt
Equations are nonlinear in tx, ty, tz
Using a different norm
dydxyxfyxf
dydxZ
tytv
Z
txtu yzxz
)(),(||),(||
min )(
222,
22
22
vu
ZZZ
vZZ
au
22
220
First, differentiate integrand with respect to Z and set to zero
),,(
min2
zyx tttg
dydxvu
zxzyyxzyxzyx tfttettdtctbtattttg 222),,( 222
where
, )(
, )(
,
, )(
,
,
2
2
dydxyuxvvf
dydxyuxvue
dydxuvd
dydxyuxvc
dydxub
dydxva
Differentiate g( tx, ty, tz ) with respect to tx, ty, tz and set to zero
cef
ebd
fda
G
||t||Gt 1 with 0
Solution is singular vector corresponding to smallest singular value
The Rotational Case min2
rot2
rot vvuu
01
012
2
xxyyv
yxxyu
zyx
zyx
v
u
xxyfy
yfxxy
z
y
x
2
2
In matrix form
u
u
TT AAA
A1
Let f=1
The General Case
Minimization of epipolar distance
or, in vector notation
min2
tr
tr
rot
rot
dydxu
v
vv
uu
min2 rrrrt d
Motion ParallaxThe difference in motion between two very close points does not depend on rotation.Can be used at depth discontinuities to obtain the direction of translation.
FOE
Motion Parallax
0
0
21,2,1
21,2,1
21
2 1
)11
)((
)11
)((
, ,
have we),(p andp pointsAt
),(
xx
yy
u
v
ZZyyvvv
ZZxxuuu
yx
vu
otrtrtr
otrtrtr
rotrot
rottr
uu
uuu
Vectors perpendicular to translational component
)|| ,||() ,(
00
00
xxyyxxyy
tr u
rotrotxxyytr vxxuyy )()( 00)|| ,||(1
00
uu
Vector component perpendicular to translational componentis only due to rotation rotation can be estimated from it.
Motion Estimation Techniques• Prazdny (1981), Burger Bhanu (1990), Nelson Aloimonos (1988), Heeger
Jepson (1992):
Decomposition of flow field into translational and rotational components. Translational flow field is radial (all vectors are emanating from (or pouring into) one point), rotational flow field is quadratic in image coordinates.
Either search in the space of rotations: remainig flow field
should be translational. Translational flow field
is evaluated by minimizing deviation from radial field:
.
Or search in the space of directions of translation: Vectors
perpendicular to translation are due to rotation only
0),(),( 00 yyxxuv
Motion Estimation Techniques
• Longuet-Higgins Prazdny (1980), Waxman (1987): Parametric model for local surface patches (planes or quadrics) solve locally for motion parameters and structure, because flow is linear in the motion parameters
(quadratic or higher order in the image coordinates)