Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney [email protected] Princeton...
-
Upload
julius-richards -
Category
Documents
-
view
213 -
download
0
Transcript of Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney [email protected] Princeton...
![Page 1: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/1.jpg)
Visual Motion Estimation
Problems & Techniques
Harpreet S. [email protected]
Princeton UniversityCOS 429 Lecture
Feb. 12, 2004
![Page 2: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/2.jpg)
Outline1. Visual motion in the Real World
2. The visual motion estimation problem
3. Problem formulation: Estimation through model-based alignment
4. Coarse-to-fine direct estimation of model parameters
5. Progressive complexity and robust model estimation
6. Multi-modal alignment
7. Direct estimation of parallax/depth/optical flow
8. Glimpses of some applications
![Page 3: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/3.jpg)
Types of Visual Motion
in the
Real World
![Page 4: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/4.jpg)
Simple Camera Motion : Pan & Tilt
Camera Does Not Change Location
![Page 5: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/5.jpg)
Apparent Motion : Pan & Tilt
Camera Moves a Lot
![Page 6: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/6.jpg)
Independent Object Motion
Objects are the FocusCamera is more or less steady
![Page 7: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/7.jpg)
Independent Object Motionwith
Camera Pan
Most common scenario for
capturing performances
![Page 8: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/8.jpg)
General Camera Motion
Large changesin
camera location & orientation
![Page 9: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/9.jpg)
Visual Motion due to Environmental Effects
Every pixel may have its own motion
![Page 10: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/10.jpg)
The Works!
General Camera & Object Motions
![Page 11: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/11.jpg)
Why is Analysis and Estimation
ofVisual Motion Important?
![Page 12: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/12.jpg)
Visual Motion Estimationas a means of extracting
Information Content in Dynamic Imagery...extract information behind pixel data...
ForegroundVs.
Background
![Page 13: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/13.jpg)
Information Content in Dynamic Imagery...extract information behind pixel data...
ForegroundVs.
Background
Extended Scene Geometry
![Page 14: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/14.jpg)
Information Content in Dynamic Imagery...extract information behind pixel data...
ForegroundVs.
Background
Extended Scene Geometry
TemporalPersistence
Layers with 2D/3DScene ModelsLayers & Mosaics
Direct method based Layered, Motion, Structure & Appearance Analysis providesCompact Representation for Manipulation & Recognition of Scene Content
Segment,Track,FingerprintMoving Objects
![Page 15: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/15.jpg)
• Pin-hole camera model
• Pure rotation of the camera
• Multiple images related through a 2D projective transformation: also called a homography
• In the special case for camera pan, with small frame-to-frame rotation, and small field of view, the frames are related through a pure image translation
An Example
A Panning Camera
![Page 16: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/16.jpg)
Pin-hole Camera Model
fZ
Y
y
Z
Yf=y fP≈p
![Page 17: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/17.jpg)
Camera Rotation (Pan)
f
Z’
Y’
y’
Z′Y′
f=y′ P′f≈p′
PR′=P′
pR′≈p′
![Page 18: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/18.jpg)
Camera Rotation (Pan)
f
Z’’
Y’’
y’’
Z ′′Y ′′
f=y ′′ P ′′f≈p ′′
PR ′′=P ′′
pR ′′≈p ′′
![Page 19: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/19.jpg)
Image Motion due to
Rotationsdoes not depend
on the depth / structure of the
scene
Verify the same for a 3D scene and 2D camera
![Page 20: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/20.jpg)
Pin-hole Camera Model
fZ
Y
y
Z
Yf=y fP≈p
![Page 21: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/21.jpg)
Camera Translation (Ty)
fZ
Y
yXX
X
X
Z′Y′
f=y′ P′f≈p′ T′+P=P′
![Page 22: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/22.jpg)
Translational Displacement
Z′Y′
f=y′
Z
Ty+Yf=y′
Z
Tyf=y-y′
Z′Y′
f=y′
Tz+Z
Yf=y′
Z
Tzyy y-- ′=′
Image Motion due to Translationis a function of
the depth of the scene
![Page 23: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/23.jpg)
![Page 24: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/24.jpg)
Sample Displacement Fields
Render scenes with various motions and plot the displacement fields
![Page 25: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/25.jpg)
Motion Field vs. Optical Flow
X
Y
Z
Tx
Ty
Tz
wx
wy
wz
P
pP’p’
Motion Field : 2D projections of 3Ddisplacement vectors due to camera and/or object motion
Optical Flow : Image displacement fieldthat measures the apparent motion ofbrightness patterns
![Page 26: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/26.jpg)
Motion Field vs. Optical Flow
Lambertian ball rotating in 3D
Motion Field ?
Optical Flow ?
Courtesy : Michael Black @ Brown.eduImage: http://www.evl.uic.edu/aej/488/
![Page 27: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/27.jpg)
Motion Field vs. Optical Flow
Stationary Lambertian ball with amoving point light source
Motion Field ?
Optical Flow ?
Courtesy : Michael Black @ Brown.eduImage : http://www.evl.uic.edu/aej/488/
![Page 28: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/28.jpg)
• Parametric motion models – 2D translation, affine, projective, 3D pose [Bergen, Anandan, et.al.’92]
• Piecewise parametric motion models– 2D parametric motion/structure layers [Wang&Adelson’93, Ayer&Sawhney’95]
• Quasi-parametric– 3D R, T & depth per pixel. [Hanna&Okumoto’91]
– Plane+parallax [Kumar et.al.’94, Sawhney’94]
• Piecewise quasi-parametric motion models– 2D parametric layers + parallax per layer [Baker et al.’98]
• Non-parametric– Optic flow: 2D vector per pixel [Lucas&Kanade’81, Bergen,Anandan et.al.’92]
A Hierarchy of Models
Taxonomy by Bergen,Anandan et al.’92
![Page 29: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/29.jpg)
Sparse/Discrete Correspondences
&
Dense Motion Estimation
![Page 30: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/30.jpg)
Discrete Methods
Feature Correlation&
RANSAC
![Page 31: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/31.jpg)
Visual Motion through Discrete Correspondences
pp′
Images may be separated by time, space, sensor types
In general, discrete correspondences are related
through a transformation
![Page 32: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/32.jpg)
Discrete Methods
Feature Correlation&
RANSAC
![Page 33: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/33.jpg)
Discrete Correspondences
• Select corner-like points• Match patches using Normalized Correlation• Establish further matches using motion model
![Page 34: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/34.jpg)
Direct Methods for Visual Motion Estimation
Employ Models of Motionand
Estimate Visual Motionthrough
Image Alignment
![Page 35: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/35.jpg)
Characterizing Direct MethodsThe What
• Visual interpretation/modeling involves spatio-temporal image representations directly– Not explicitly represented discrete features like
corners, edges and lines etc.
• Spatio-temporal images are represented as outputs of symmetric or oriented filters.
• The output representations are typically dense, that is every pixel is explained,– Optical flow, depth maps.– Model parameters are also computed.
![Page 36: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/36.jpg)
Direct Methods : The How Alignment of spatio-temporal images is a means of obtaining :
Dense Representations, Parametric Models
![Page 37: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/37.jpg)
Direct Method based Alignment
![Page 38: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/38.jpg)
Formulation of Direct Model-based Image Alignment[Bergen,Anandan et al.’92]
)p(I1 )p(I2
p)p(up
Model image transformation as :
)());(()( 112 pIpupIpI
Images separated by
time, space, sensor types
BrightnessConstancy
![Page 39: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/39.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Model image transformation as :
))Θ;p(up(I)p(I 12
Images separated by
time, space, sensor types
Reference Coordinate
System
![Page 40: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/40.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Model image transformation as :
))Θ;p(up(I)p(I 12
Images separated by
time, space, sensor types
Reference Coordinate
System
Generalized pixel
Displacement
![Page 41: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/41.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Model image transformation as :
))Θ;p(up(I)p(I 12
Images separated by
time, space, sensor types
Reference Coordinate
System
Generalized pixel
Displacement
ModelParameters
![Page 42: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/42.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Model image transformation as :
))Θ;p(up(I)p(I 12
Images separated by
time, space, sensor types
Reference Coordinate
System
Generalized pixel
Displacement
ModelParameters
![Page 43: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/43.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Compute the unknown parameters and correspondenceswhile aligning images using optimization :
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
What all can be varied ?
Filtered ImageRepresentations(to account for
Illumination changes,Multi-modalities)
![Page 44: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/44.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Compute the unknown parameters and correspondenceswhile aligning images using optimization :
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
What all can be varied ?
Filtered ImageRepresentations(to account for
Illumination changes,Multi-modalities)
ModelParameters
![Page 45: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/45.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Compute the unknown parameters and correspondenceswhile aligning images using optimization :
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
What all can be varied ?
Filtered ImageRepresentations(to account for
Illumination changes,Multi-modalities)
ModelParameters
Measuringmismatches
(SSD, Correlations)
![Page 46: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/46.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Compute the unknown parameters and correspondenceswhile aligning images using optimization :
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
What all can be varied ?
Filtered ImageRepresentations(to account for
Illumination changes,Multi-modalities)
ModelParameters
Measuringmismatches
(SSD, Correlations)
OptimizationFunction
![Page 47: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/47.jpg)
Formulation of Direct Model-based Image Alignment
)p(I1 )p(I2
p)p(up
Compute the unknown parameters and correspondenceswhile aligning images using optimization :
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
What all can be varied ?
Filtered ImageRepresentations(to account for
Illumination changes,Multi-modalities)
ModelParameters
Measuringmismatches
(SSD, Correlations)
OptimizationFunction
![Page 48: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/48.jpg)
• Parametric motion models – 2D translation, affine, projective, 3D pose [Bergen, Anandan, et.al.’92]
• Piecewise parametric motion models– 2D parametric motion/structure layers [Wang&Adelson’93, Ayer&Sawhney’95]
• Quasi-parametric– 3D R, T & depth per pixel. [Hanna&Okumoto’91]
– Plane+parallax [Kumar et.al.’94, Sawhney’94]
• Piecewise quasi-parametric motion models– 2D parametric layers + parallax per layer [Baker et al.’98]
• Non-parametric– Optic flow: 2D vector per pixel [Lucas&Kanade’81, Bergen,Anandan et.al.’92]
A Hierarchy of Models
Taxonomy by Bergen,Anandan et al.’92
![Page 49: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/49.jpg)
Plan : This Part
• First present the generic normal equations.
• Then specialize these for a projective transformation.
• Sidebar into backward image warping.
• SSD and M-estimators.
![Page 50: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/50.jpg)
An Iterative Solution of Model Parameters
[Black&Anandan’94 Sawhney’95]
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
• Given a solution )m(Θ at the mth iteration, find Θδ by solving :
l i i k
ii
i
il
l
i
k
i
i
i rr
r
rrr
r
r
)()(
k
iw
• iw is a weight associated with each measurement.
![Page 51: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/51.jpg)
An Iterative Solution of Model Parameters
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
• In particular for Sum-of-Square Differences :
l i i k
iil
l
i
k
i rr
rr
k
2
2
2 r
SSD
• We obtain the standard normal equations:
• Other functions can be used for robust M-estimation…
![Page 52: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/52.jpg)
How does this work for images ? (1)
i
ir ,2
1min 2 ));(()( 12 iiii pupIpIr
ip
• First, warp
• Given an initial guess)k(Ρ
)(1 ipI towards )(2 ipI
• Let their be a 2D projective transformation between the two images:
ii pp Ρ≈′
![Page 53: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/53.jpg)
How does this work for images ? (2)
i
ir ,2
1min 2 ));(()( 12 iiii pupIpIr
ip
( ) )p(ΡI)p(I)(pI wk11
ww1 =′=
)p(I1 )p(I2
p)p(up
)(pI ww1
( ) wk pΡ
![Page 54: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/54.jpg)
How does this work for images ? (3)
i
ir ,2
1min 2 ));(()( 12 iiii pupIpIr
ip( ) )p(ΡI)p(I)(pI wk11
ww1 =′=
)p(I1 )p(I2
p)p(up
)(pI ww1
( ) wk pΡ
:)(pI ww1
Represents image 1 warped towards the reference image 2,Using the current set of parameters
![Page 55: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/55.jpg)
How does this work for images ? (4)
• The residual transformation between the warped image and the reference image is modeled as:
));p(pp(I)p(Ir iiiw
1i2i Θδδ= www --
Wherei
wi D]p[Ip +≈
=
032d31d
23d22d21d
13d12d11d
D
![Page 56: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/56.jpg)
How does this work for images ? (5)
• The residual transformation between the warped image and the reference image is modeled as:
))D;p(pp(I)p(I=r iiiw
1i2iwww -- δ
d|d
pI;0))(p(pI)(pI 0D
wiw
iw
iw
i2
T
=∂
∂∇--≈
1ydxd
d)yd(1xd1ydxd
dyd)xd(1
p
3231
232221
3231
131211
w
2
2
0D
w
yxy1yx000
xyx0001yx|
D
p
![Page 57: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/57.jpg)
How does this work for images ? (6)
)I(p-∇∇≈r ii δ= d|pI 0D
wiD
wi
T
i
ir ,2
1min 2
)I(p∇∇∇ ∇∇ iii
∑∑ δ= wi
TD
wiD
wi
wi
wi
TD pdpIIp
TT
gHd =
D][IΡΡ (k)1)(k
So now we can solve for the model parameters while aligning images iteratively using warping and Levenberg-Marquat style optimization
![Page 58: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/58.jpg)
Sidebar : Backward Warping• Note that we have used backward warping in the direct
alignment of images.
• Backward warping avoids holes.
• Image gradients are estimated in the warped coordinate system.
Target Image : EmptySource Image : Filled
wpΡp =′Bilinear Warp
![Page 59: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/59.jpg)
Sidebar : Backward Warping• Note that we have used backward warping in the direct
alignment of images.
• Backward warping avoids holes.
• Image gradients are estimated in the warped coordinate system.
Target Image : EmptySource Image : Filled
wpΡp =′Bicubic Warp
![Page 60: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/60.jpg)
delay
estimateglobal shift
warp
)(tI
)1(ˆ tI
)(td
)(td
I (t 1)
Iterative Alignment : Result
![Page 61: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/61.jpg)
How to handle Large Transformations ?
[Burt,Adelson’81]
• A hierarchical framework for fast algorithms
• A wavelet representation for compression, enhancement, fusion
• A model of human vision
Gaussian
Laplacian
PyramidProcessing
![Page 62: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/62.jpg)
Iterative Coarse-to-fine Model-based Image Alignment Primer
p
2
ΘΘ))u(p;(pI(p)I( 21min )
{ R, T, d(p) }{ H, e, k(p) }
{ dx(p), dy(p) }
d(p)
Warper -
![Page 63: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/63.jpg)
• Coarse levels reduce search.
• Models of image motion reduce modeling complexity.
• Image warping allows model estimation without discrete feature extraction.
• Model parameters are estimated using iterative non-linear optimization.
• Coarse level parameters guide optimization at finer levels.
Pyramid-based Direct Image Alignment Primer
![Page 64: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/64.jpg)
Application : Image/Video Mosaicing
• Direct frame-to-frame image alignment.
• Select frames to reduce the number of frames & overlap.
• Warp aligned images to a reference coordinate system.
• Create a single mosaic image.
• Assumes a parametric motion model.
![Page 65: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/65.jpg)
Princeton Chapel Video Sequence54 frames
Video Mosaic ExampleVideoBrush’96
![Page 66: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/66.jpg)
Unblended Chapel Mosaic
![Page 67: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/67.jpg)
• Chips are images.
• May or may not be captured from known locations of the camera.
Image Mosaics
![Page 68: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/68.jpg)
Output Mosaic
![Page 69: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/69.jpg)
Handling Moving Objects in 2D Parametric Alignment &
Mosaicing
![Page 70: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/70.jpg)
Generalized M-Estimation
i
iΘ
),σ;r(ρmin ));(()( 12 iiii pupIpIr
• Given a solution )m(Θ at the mth iteration, find Θδ by solving :
l i i k
ii
i
il
l
i
k
i
i
i rr
r
rrr
r
r
)()(
k
iw
• iw is a weight associated with each measurement.Can be varied to provide robustness to outliers.
Choices of the );r( i σρ function:
2SS
σ
1
r
)r(ρ
222
2GM
)rσ(
σ2
r
)r(ρ
2
2
SS σ2
rρ
22
22
GM σr1
σrρ
![Page 71: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/71.jpg)
Optimization Functions & their Corresponding Weight Plots
Geman-Mclure Sum-of-squares
![Page 72: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/72.jpg)
With Robust Functions Direct Alignment Works
for Non-dominant Moving Objects Too
Original two frames Background Alignment
![Page 73: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/73.jpg)
Object Deletion with Layers
Video Stream with deleted moving objectOriginal Video
![Page 74: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/74.jpg)
Optic Flow Estimation
))D;p(pp(I)p(I=r iiiw
1i2iwww -- δ
pI;0))(p(pI)(pITw
iw
iw
i2 δ∇--≈
[ ] I)(pI)(pIy
xII w
iw
i2wy
wx δ
δ
δ=≈ -
Gradient Direction Flow Vector
![Page 75: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/75.jpg)
Normal Flow ConstraintAt a single pixel, brightness constraint:
0IvIuI tyx =++
Normal FlowI
I- t
∇
![Page 76: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/76.jpg)
![Page 77: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/77.jpg)
![Page 78: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/78.jpg)
![Page 79: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/79.jpg)
![Page 80: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/80.jpg)
Computing Optical Flow:Discretization
• Look at some neighborhood N:
0
0),(),(
want
want
N),(
T
bAv
v jiIjiIji
t
0
0),(),(
want
want
N),(
T
bAv
v jiIjiIji
t
),(
),(
),(
),(
),(
),(
22
11
22
11
nnt
t
t
nn jiI
jiI
jiI
jiI
jiI
jiI
bA
),(
),(
),(
),(
),(
),(
22
11
22
11
nnt
t
t
nn jiI
jiI
jiI
jiI
jiI
jiI
bA
![Page 81: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/81.jpg)
Computing Optical Flow:Least Squares
• In general, overconstrained linear system
• Solve by least squares
bAAAv
bAvAA
bAv
T1T
TT
want
)(
)(
0
bAAAv
bAvAA
bAv
T1T
TT
want
)(
)(
0
![Page 82: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/82.jpg)
Computing Optical Flow:Stability
• Has a solution unless C = ATA is singular
Ny
Nyx
Nyx
Nx
nn
nn
III
III
jiI
jiI
jiI
jiIjiIjiI
2
2
22
11
2211
T
),(
),(
),(
),(),(),(
C
C
AAC
Ny
Nyx
Nyx
Nx
nn
nn
III
III
jiI
jiI
jiI
jiIjiIjiI
2
2
22
11
2211
T
),(
),(
),(
),(),(),(
C
C
AAC
![Page 83: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/83.jpg)
Computing Optical Flow:Stability
• Where have we encountered C before?• Corner detector!• C is singular if constant intensity or edge• Use eigenvalues of C:
– to evaluate stability of optical flow computation– to find good places to compute optical flow
(finding good features to track)– [Shi-Tomasi]
![Page 84: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/84.jpg)
Example of Flow Computation
![Page 85: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/85.jpg)
Example of Flow Computation
![Page 86: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/86.jpg)
Example of Flow Computation
But this in general is not the motion field
![Page 87: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/87.jpg)
Motion Field = Optical Flow ?
From brightness constancy, normal flow:E
E-
E
v)E(v t
T
n ∇∇∇
==
Motion field for a Lambertian scene:
E
)xn(Iv
T
∇=∴
ωρΔ
nIE Tρ= )xn(IEvE Tt
T ωρ=+∇xnω=dt
dn
Points with high spatial gradient are the locationsAt which the motion field can be best estimated
By brightness constancy (the optical flow)
![Page 88: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/88.jpg)
Motion Illusions in
Human Vision
![Page 89: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/89.jpg)
Aperture Problem
• Too big:confused bymultiple motions
• Too small:only get motionperpendicularto edge
![Page 90: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/90.jpg)
Ouchi Illusion
The Ouchi illusion, illustrated above, is an illusion named after its inventor,
Japanese artist Hajime Ouchi. In this illusion, the central disk seems to float above the checkered background when
moving the eyes around while viewing the figure. Scrolling the image horizontally or vertically give a much stronger effect.
The illusion is caused by random eye movements, which are independent in the horizontal and vertical directions.
However, the two types of patterns in the figure nearly eliminate the effect of the eye movements parallel to each type
of pattern. Consequently, the neurons stimulated by the disk convey the signal that the disk jitters due to the horizontal
component of the eye movements, while the neurons stimulated by the background convey the signal that movements
are due to the independent vertical component. Since the two regions jitter independently, the brain interprets the regions
as corresponding to separate independent objects (Olveczky et al. 2003).
http://mathworld.wolfram.com/OuchiIllusion.html
![Page 91: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/91.jpg)
Akisha Kitakao
http://www.ritsumei.ac.jp/~akitaoka/saishin-e.html
![Page 92: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/92.jpg)
Rotating Snakes
![Page 93: Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney hsawhney@sarnoff.com Princeton University COS 429 Lecture Feb. 12, 2004.](https://reader035.fdocuments.in/reader035/viewer/2022070410/56649ea85503460f94bac8fb/html5/thumbnails/93.jpg)