Post on 22-Feb-2016
description
),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):
Brightness Constancy Equation:
The Brightness Constraint
),(),( ),(),( yxyx vyuxIyxJ
Where: ),(),( yxJyxIIt
-=
0 tyx IvIuI
Each pixel provides 1 equation in 2 unknowns (u,v). Insufficient info.
Another constraint: Global Motion Model Constraint
The 2D/3D Dichotomy
Image motion =
Camera induced motion =
+ Independent motions =
Camera motion+
Scene structure
+Independent motions
2D techniques
3D techniques Singularities in
“2D scenes”
Do not model
“3D scenes”
Requires prior model selection
Global Motion Models2D Models:• Affine• Quadratic• Homography (Planar projective transform)
3D Models:• Rotation, Translation, 1/Depth • Instantaneous camera motion models• Essential/Fundamental Matrix• Plane+Parallax
0)()( 654321 tyx IyaxaaIyaxaaI
Example: Affine Motion
Substituting into the B.C. Equation:
yaxaayxvyaxaayxu
654
321
),(),(
==
Each pixel provides 1 linear constraint in 6 global unknowns
0 tyx IvIuI
(minimum 6 pixels necessary)
2 = tyx IyaxaaIyaxaaIaErr )()()( 654321
Least Square Minimization (over all pixels):
Every pixel contributes Confidence-weighted regression
Example: Affine Motion 2 = tyx IyaxaaIyaxaaIaErr )()()( 654321
Differentiating w.r.t. a1 , …, a6 and equating to zero
22222
22222
222
22222
22222
222
yyyyxyxyx
yyyyxyxyx
yyyyxyxyx
yxyxyxxxx
yxyxyxxxx
yxyxyxxxx
IyxyIyIIIyIxyIIyIxyIIxxIIxyIIIxIxIyIxIIIyIIxIII
IIyIxyIIyIIyxyIyIIxyIIIxIxIxyIIxxIIyIIxIIIyIxII
=
6
5
4
3
2
1
aaaaaa
------
ty
ty
ty
tx
tx
tx
IyIIxIIIIyIIxIII
6 linear equations in 6 unknowns:
image Iimage J
aJwwarp refine
a aΔ+
Pyramid of image J Pyramid of image I
image Iimage J
Coarse-to-Fine Estimation
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
0 tyx IvIuI ==> small u and v ...
Parameter propagation: )2()2(2),(2)2,2(
)2()2(2),(2)2,2(
654
321
yaxaayxvyxvyaxaayxuyxu
====
)(2)(2
654
321
yaxaayaxaa
==
Quadratic – instantaneous approximation to planar motion
Other 2D Motion Models
287654
82
7321
yqxyqyqxqqv
xyqxqyqxqqu
=
=
yyvxxu
yhxhhyhxhh
pHpHy
yhxhhyhxhh
pHpHx
pHHp
PHHP
PPp
Z
-=-=
==
==
===
',' and
'
'
'''
987
654
3
2
987
321
3
1
33Projective – exact planar motion
(Homography H)
0)( 321 -- pHyIxIIpHIpHI yxtyx 0 tyx IvIuI
Panoramic Mosaic ImageOriginal video clip
Generated Mosaic image
Alignment accuracy (between a pair of frames): error < 0.1 pixel
Original
Outliers
Original
Synthesized
Video Removal
ORIGINAL ENHANCED
Video Enhancement
Direct Methods: Methods for motion and/or shape estimation, which recover the unknown parameters directly from measurable image quantities at each pixel in the image.
Minimization step: Direct methods: Error measure based on dense measurable image quantities(Confidence-weighted regression; Exploits all available information)
Feature-based methods: Error measure based on distances of a sparse set of distinct feature matches.
Image gradients The descriptor(4x4 array of 8-bin histograms)
– Compute gradient orientation histograms of several small windows (128 values for each point)
– Normalize the descriptor to make it invariant to intensity change– To add Scale & Rotation invariance:
Determine local scale (by maximizing DoG in scale and in space), local orientation as the dominant gradient direction.
Example: The SIFT Descriptor
D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004
• Compute descriptors in each image Find descriptors matches across images
Estimate transformation between the pair of images.• In case of multiple motions:
Use RANSAC (Random Sampling and Consensus) to compute Affine-transformation / Homography / Essential-Matrix / etc.
Benefits of Direct Methods
• High subpixel accuracy.• Simultaneously estimate matches + transformation
Do not need distinct features.• Strong locking property.
Limitations
• Limited search range (up to ~10% of the image size).
• Brightness constancy assumption.
Video Indexing and Editing
The 2D/3D Dichotomy
Image motion =
Camera induced motion =
+ Independent motions =
Camera motion+
Scene structure
+Independent motions
2D techniques
3D techniques Singularities in
“2D scenes”
Do not model
“3D scenes”
A camera-centric coordinate system (R,T,Z)
The Plane+Parallax Decomposition
Original Sequence Plane-Stabilized Sequence
The residual parallax lies on a radial (epipolar) field: wp'
pepipole
p'
Benefits of the P+P Decomposition
• Eliminates effects of rotation
• Eliminates changes in camera parameters / zoom
• Camera parameters: Need to estimate only epipole. (gauge ambiguity: unknown scale of epipole)
• Image displacements: Constrained to lie on radial lines (1-D search problem)
A result of aligning an existing structure in the image.
1. Reduces the search space:
Remove global component which dilutes information !
Translation or pure rotation ???
Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Focus on relevant portion of info
Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Shape = Fluctuations relative to a planar surface in the scene
STAB_RUG SEQ
- fewer bits, progressive encoding
Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Shape = Fluctuations relative to a planar surface in the scene• Height vs. Depth (e.g., obstacle avoidance)
• A compact representation
global (100)component
local [-3..+3]component
total distance [97..103]
camera center scene
• Appropriate units for shape
• Start with 2D estimation (homography).
• 3D info builds on top of 2D info.
3. Stratified 2D-3D Representation:
Avoids a-priori model selection.
Benefits of the P+P Decomposition
Original sequence Plane-aligned sequence Recovered shape
Dense 3D Reconstruction(Plane+Parallax)
Dense 3D Reconstruction(Plane+Parallax)
Original sequence
Plane-aligned sequence
Recovered shape
Original sequence Plane-aligned sequence
Recovered shape
Dense 3D Reconstruction(Plane+Parallax)
Brightness Constancy constraint
P+P Correspondence Estimation
The intersection of the two line constraints uniquely defines the displacement.
1. Eliminating Aperture Problem
Epipolar line
epipole
p
0= TYX IvIuI
other epipolar line
Epipolar line
Multi-Frame vs. 2-Frame Estimation
The two line constraints are parallel ==> do NOT intersect
1. Eliminating Aperture Problem
p
0= TYX IvIuI
anotherepipole
epipole
Brightness Constancy constra
int
The other epipole resolves the ambiguity !