Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and...
-
Upload
laila-pearce -
Category
Documents
-
view
216 -
download
0
Transcript of Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and...
![Page 1: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/1.jpg)
Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj
Go With The FlowA New Manifold Modeling and Learning Framework for Image Ensembles
Aswin C. SankaranarayananRice University
![Page 2: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/2.jpg)
Sensor Data Deluge
![Page 3: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/3.jpg)
Concise Models
• Efficient processing / compression requires concise representation
• Sparsity of an individual image
pixels largewaveletcoefficients
(blue = 0)
![Page 4: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/4.jpg)
Concise Models
• Efficient processing / compression requires concise representation
• Our interest in this talk: Collections of images
![Page 5: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/5.jpg)
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
![Page 6: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/6.jpg)
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
• Image articulation manifold
![Page 7: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/7.jpg)
Image Articulation Manifold• N-pixel images:
• K-dimensional articulation space
• Thenis a K-dimensional manifoldin the ambient space
• Very concise model
articulation parameter space
![Page 8: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/8.jpg)
Smooth IAMs• N-pixel images:
• Local isometry: image distance parameter space distance
• Linear tangent spacesare close approximationlocally
articulation parameter space
![Page 9: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/9.jpg)
Smooth IAMs• N-pixel images:
• Local isometry: image distance parameter space distance
• Linear tangent spacesare close approximationlocally
articulation parameter space
![Page 10: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/10.jpg)
Ex: Manifold Learning
LLEISOMAPLEHEDiff. Geo …
• K=1rotation
![Page 11: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/11.jpg)
Ex: Manifold Learning
• K=2rotation and scale
![Page 12: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/12.jpg)
Theory/Practice Disconnect: Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?Isometry ?
![Page 13: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/13.jpg)
Theory/Practice Disconnect: Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?Isometry ?
![Page 14: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/14.jpg)
Failure of Tangent Plane Approx.
• Ex: cross-fading when synthesizing / interpolating images that should lie on manifold
Input Image Input Image
Geodesic Linear path
![Page 15: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/15.jpg)
Failure of Local Isometry
• Ex: translation manifold
all blue imagesare equidistantfrom the red image
• Local isometry
– satisfied only when sampling is dense
0 20 40 60 80 1000
0.5
1
1.5
2
2.5
3
3.5
Translation in [px]
Euc
lidea
n di
stan
ce
![Page 16: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/16.jpg)
Tools for manifold processing
Geodesics, exponential maps, log-maps, Riemannian metrics, Karcher means, …
Smooth diff manifold
Algebraic manifolds Data manifolds
LLE, kNN graphs
Point cloud model
![Page 17: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/17.jpg)
The concept ofTransport operators
))(()( xfIxIref
Beyond point cloud model for image manifolds
refIfI
Example
![Page 18: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/18.jpg)
Example: Translation
• 2D Translation manifold
barring boundary related issues
• Set of all transport operators =
• Beyond a point cloud model– Action of the articulation is more accurate and meaningful
1I
11101 )( ),()( xxfxIxI
0I
2R
![Page 19: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/19.jpg)
Optical Flow
• Generalizing this idea: Pixel correspondances
• Idea:OF between two images is a natural and accurate transport operator
(Figures from Ce Liu’s optical flow page)
OFfrom I1 to I2
I1 and I2
![Page 20: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/20.jpg)
Optical Flow Transport
0
1 2
IAM
2I1
I0I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
![Page 21: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/21.jpg)
Optical Flow Transport
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
• Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)
![Page 22: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/22.jpg)
OFM is Smooth (Rotation)
00 30 60 30 60
-80 -60 -40 -20 0 20 40 60 800
50
100
150
200
Articulation in [ ]
Inte
nsity
-80 -60 -40 -20 0 20 40 60 80-20
-10
0
10
20
Articulation in [ ]
Opt
ical
flow
vx i
n pi
xels
Articulation θ in [⁰]
Inte
nsi
ty I(θ
)O
p. flow
v(θ
)Pixel intensityat 3 points
Flow (nearly linear)
![Page 23: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/23.jpg)
Main results
• Local model at each
• Each point on the OFM defines a transport operator– Each transport operator maps
to one of its neighbors
• For a large class of articulations, OFMs are smooth and locally isometric– Traditional manifold processing
techniques work on OFMs
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
0I
0I
![Page 24: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/24.jpg)
Linking it all together
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
0e
1e 2e
Nonlinear dim. reduction
The non-differentiablity does not dissappear --- it is embedded in the mapping from OFM to the IAM.
However, this is a known map
))(()( such that },{ xfIxIIfI refref
![Page 25: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/25.jpg)
The Story So Far…
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
Tangent space at 0I
Articulations
0
1 2
2I1
I0I IAM
![Page 26: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/26.jpg)
Input Image Input Image
Geodesic Linear pathIAM
OFM
![Page 27: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/27.jpg)
OFM Synthesis
![Page 28: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/28.jpg)
ISOMAP embedding error for OFM and IAM
2D rotations
Reference image
Manifold Learning
![Page 29: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/29.jpg)
Manifold Learning
-6 -4 -2 0 2 4 6-5
-4
-3
-2
-1
0
1
2
3
4
5Two-dimensional Isomap embedding (with neighborhood graph).
Embedding of OFM
2D rotations
Reference image
![Page 30: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/30.jpg)
Data196 images of two bears moving linearlyand independently
IAM OFM
TaskFind low-dimensional embedding
OFM Manifold Learning
![Page 31: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/31.jpg)
IAM OFM
Data196 images of a cup moving on a plane
Task 1Find low-dimensional embedding
Task 2Parameter estimation for new images(tracing an “R”)
OFM ML + Parameter Estimation
![Page 32: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/32.jpg)
• Point on the manifold such that the sum of geodesic distances to every other point is minimized
• Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al]
Karcher Mean
10 images from an IAM
ground truth KM OFM KM linear KM
![Page 33: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/33.jpg)
Manifold Charting
• Goal: build a generative model for an entire IAM/OFM based on a small number of base images
• Ex: cube rotating about axis. All cube images can be representing using 4 reference images + OFMs
• Many applications– selection of target templates for classification– “next-view” selection for adaptive sensing applications
Greedy charting
Optimal charting
0 90 180 180 90
IAM
![Page 34: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/34.jpg)
Summary
• IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints
• But practical IAMs are non-differentiable– IAM-based algorithms have not lived up to their promise
• Optical flow manifolds (OFMs)– smooth even when IAM is not– OFM ~ nonlinear tangent space– support accurate image synthesis, learning, charting, …
• Barely discussed here: OF enables the safe extension of differential geometry concepts– Log/Exp maps, Karcher mean, parallel transport, …
![Page 35: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/35.jpg)
Open Questions
• Our treatment is specific to image manifolds under
brightness constancy
• What are the natural transport operators for other data manifolds?
dsp.rice.edu
![Page 36: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/36.jpg)
Related Work
• Analytic transport operators– transport operator has group structure [Xiao and Rao 07]
[Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08]
– non-linear analytics [Dollar et al 06]
– spatio-temporal manifolds [Li and Chellappa 10]
– shape manifolds [Klassen et al 04]
• Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups)
• In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations
![Page 37: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/37.jpg)
Limitations
• Brightness constancy– Optical flow is no longer meaningful
• Occlusion– Undefined pixel flow in theory, arbitrary flow estimates in
practice– Heuristics to deal with it
• Changing backgrounds etc.– Transport operator assumption too strict– Sparse correspondences ?
![Page 38: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/38.jpg)
![Page 39: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/39.jpg)
Open Questions
• Theorem: random measurements stably embed aK-dim manifoldwhp [B, Wakin, FOCM ’08]
• Q: Is there an analogousresult for OFMs?
![Page 40: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/40.jpg)
Image Articulation Manifold
• Linear tangent space at is K-dimensional
– provides a mechanism to transport along manifold
– problem: since manifold is non-differentiable, tangent approximation is poor
• Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process
Tangent space at 0I
Articulations
0
1 2
2I1
I0I IAM
![Page 41: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/41.jpg)
![Page 42: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/42.jpg)
OFM Implementation details
Reference Image
![Page 43: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/43.jpg)
Pairwise distances and embedding
-5 0 5
-4
-2
0
2
4
100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
![Page 44: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/44.jpg)
-5 0 5
-4
-2
0
2
4100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
Flow Embedding
![Page 45: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/45.jpg)
![Page 46: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/46.jpg)
Occlusion
• Detect occlusion using forward-backward flow reasoning
• Remove occluded pixel computations
• Heuristic --- formal occlusion handling is hard
Occluded
![Page 47: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/47.jpg)
![Page 48: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/48.jpg)
History of Optical Flow
• Dark ages (<1985)– special cases solved– LBC an under-determined set of linear equations
• Horn and Schunk (1985) – Regularization term: smoothness prior on the flow
• Brox et al (2005)– shows that linearization of brightness constancy (BC) is
a bad assumption– develops optimization framework to handle BC directly
• Brox et al (2010), Black et al (2010), Liu et al (2010)– practical systems with reliable code
![Page 49: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/49.jpg)
![Page 50: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649c7e5503460f94933e51/html5/thumbnails/50.jpg)