Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes...
-
Upload
ralph-clinton-matthews -
Category
Documents
-
view
216 -
download
0
Transcript of Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes...
![Page 1: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/1.jpg)
Markov Random Fields,Graph Cuts,
Belief Propagation
Computational Photography
Connelly Barnes
Slides from Bill Freeman
![Page 2: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/2.jpg)
Stereo Correspondence Problem
L R
d
Squared difference,
(L[x] – R[x-d])^2,for some x.
Showing local disparity evidence vectors for a set of neighboring positions, x.
x
d
![Page 3: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/3.jpg)
Super-resolution image synthesis
How to choose which selection of high resolution patches best fits together? Ignoring which patch fits well with which gives this result for the high frequency components of an image:
![Page 4: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/4.jpg)
Things we want to be able to articulate in a spatial prior
• Favor neighboring pixels having the same state (state, meaning: estimated depth, stereo disparity, group segment membership)
• Favor neighboring nodes have compatible states (a patch at node i should fit well with selected patch at node j).
• But encourage state changes to occur at certain places (like regions of high image gradient).
![Page 5: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/5.jpg)
Graphical models: tinker toys to build complex probability distributions
http://mark.michaelis.net/weblog/2002/12/29/Tinker%20Toys%20Car.jpg
• Circles represent random variables.• Lines represent statistical dependencies.• There is a corresponding equation that gives P(x1, x2, x3, y, z),
but often it’s easier to understand things from the picture.• These tinker toys for probabilities let you build up, from simple,
easy-to-understand pieces, complicated probability distributions involving many variables.
x1 x2 x3
y z
![Page 6: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/6.jpg)
Steps in building and using graphical models
• First, define the function you want to optimize. Note the two common ways of framing the problem– In terms of probabilities. Multiply together component terms,
which typically involve exponentials.– In terms of energies. The log of the probabilities. Typically add
together the exponentiated terms from above.
• The second step: optimize that function. For probabilities, take the mean or the max (or use some other “loss function”). For energies, take the min.
• 3rd step: in many cases, you want to learn the function from the 1st step.
![Page 7: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/7.jpg)
Define model parameters
1
1
1
![Page 8: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/8.jpg)
A more general compatibility matrix (values shown as grey scale)
![Page 9: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/9.jpg)
One type of graphical model: Markov Random Fields
• Allows rich probabilistic models for images.• But built in a local, modular way. Learn local
relationships, get global effects out.
“Hidden” variables,e.g. depth at each pixel
“Observed” variables,e.g. pixel intensity
![Page 10: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/10.jpg)
MRF nodes as pixels
Winkler, 1995, p. 32
![Page 11: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/11.jpg)
MRF nodes as patches
image patches
(xi, yi)
(xi, xj)
image
scene
scene patches
![Page 12: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/12.jpg)
Network joint probability
scene
image
Scene-scenecompatibility
functionneighboringscene nodes
local observations
Image-scenecompatibility
function
i
iiji
ji yxxxZ
yxP ),(),(1
),(,
![Page 13: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/13.jpg)
In order to use MRFs:
• Given observations y, and the parameters of the MRF, how infer the hidden variables, x?
• How to learn the parameters of the MRF?
![Page 14: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/14.jpg)
Outline of MRF section
• Inference in MRF’s.– Iterated conditional modes (ICM)– Gibbs sampling, simulated annealing– Belief propagation– Graph cuts
• Applications of inference in MRF’s.
![Page 15: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/15.jpg)
Iterated conditional modes
• Initialize nodes (e.g. random)
• For each node:– Condition on all the neighbors– Find the mode– Repeat.
Described in: Winkler, 1995. Introduced by Besag in 1986.
![Page 16: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/16.jpg)
Winkler, 1995
![Page 17: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/17.jpg)
Outline of MRF section
• Inference in MRF’s.– Iterated conditional modes (ICM)– Gibbs sampling, simulated annealing– Belief propagation– Graph cuts
• Applications of inference in MRF’s.
![Page 18: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/18.jpg)
Gibbs Sampling and Simulated Annealing
• Gibbs sampling: – A way to generate random samples from a (potentially
very complicated) probability distribution.
• Simulated annealing:– A schedule for modifying the probability distribution so
that, at “zero temperature”, you draw samples only from the MAP solution.
Reference: Geman and Geman, IEEE PAMI 1984.
![Page 19: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/19.jpg)
Simulated Annealing Visualization
https://en.wikipedia.org/wiki/Simulated_annealing
![Page 20: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/20.jpg)
Gibbs Sampling
x1
x2
),,,|(~ )()(3
)(21
)1(1
tK
ttt xxxxx
),,,|(~ )()(3
)1(12
)1(2
tK
ttt xxxxπx
),,|(~ )1(1
)1(1
)1(
tK
tK
tK xxxx
Slide by Ce Liu
![Page 21: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/21.jpg)
Gibbs sampling and simulated annealing
Simulated annealing as you gradually lower the “temperature” of the probability distribution ultimately giving zero probability to all but the MAP estimate.
What’s good about it: finds global MAP solution.
What’s bad about it: takes forever. Gibbs sampling is in the inner loop…
![Page 22: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/22.jpg)
Gibbs sampling and simulated annealing
You can find the mean value (MMSE estimate) of a variable by doing Gibbs sampling and averaging over the values that come out of your sampler.
You can find the MAP value of a variable by doing Gibbs sampling and gradually lowering the temperature parameter to zero.
![Page 23: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/23.jpg)
Outline of MRF section
• Inference in MRF’s.– Iterated conditional modes (ICM)– Gibbs sampling, simulated annealing– Belief propagation– Graph cuts
• Applications of inference in MRF’s.
![Page 24: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/24.jpg)
),,,,,(sumsummean 3213211321
yyyxxxPxxxx
MMSE
y1
Derivation of belief propagation
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3
Goal: Compute marginal probabilities of x1
(probabilities of x1 alone, given the observed variables) From these, compute MMSE, Minimum Mean Squared Error estimator of x1, which is just E(x1 | observed variables)
“Hidden variables”,e.g. depth in scene
“Observed variables”,e.g. image intensity
BP Tutorial: http://graphics.tu-bs.de/static/teaching/seminars/ss13/CG/papers/Klose02.pdf
![Page 25: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/25.jpg)
),,,,,(sumsummean 3213211321
yyyxxxPxxxx
MMSE
y1
Derivation of belief propagation
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3
![Page 26: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/26.jpg)
),(),(sum
),(),(sum
),(mean
),(),(
),(),(
),(sumsummean
),,,,,(sumsummean
3233
2122
111
3233
2122
111
3213211
3
2
1
321
321
xxyx
xxyx
yxx
xxyx
xxyx
yxx
yyyxxxPx
x
x
xMMSE
xxxMMSE
xxxMMSE
The posterior factorizes
y1
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3
![Page 27: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/27.jpg)
Propagation rules
y1
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3),(),(sum
),(),(sum
),(mean
),(),(
),(),(
),(sumsummean
),,,,,(sumsummean
3233
2122
111
3233
2122
111
3213211
3
2
1
321
321
xxyx
xxyx
yxx
xxyx
xxyx
yxx
yyyxxxPx
x
x
xMMSE
xxxMMSE
xxxMMSE
![Page 28: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/28.jpg)
Propagation rules
y1
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3
),(),(sum
),(),(sum
),(mean
3233
2122
111
3
2
1
xxyx
xxyx
yxx
x
x
xMMSE
)( ),( ),(sum)( 23222211
21
2
xMyxxxxMx
![Page 29: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/29.jpg)
Propagation rules
y1
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3
),(),(sum
),(),(sum
),(mean
3233
2122
111
3
2
1
xxyx
xxyx
yxx
x
x
xMMSE
)( ),( ),(sum)( 23222211
21
2
xMyxxxxMx
![Page 30: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/30.jpg)
Belief propagation: the nosey neighbor rule
“Given everything that I know, here’s what I think you should think”
(Given the probabilities of my being in different states, and how my states relate to your states, here’s what I think the probabilities of your states should be)
![Page 31: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/31.jpg)
Belief propagation messages
jii =
ijNk
jkjji
xi
ji xMxxxM
j \)(ij )(),( )(
j
To send a message: Multiply together all the incoming messages, except from the node you’re sending to,then multiply by the compatibility matrix and marginalize over the sender’s states.
A message: can be thought of as a set of weights on each of your possible states
![Page 32: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/32.jpg)
Beliefs
j
)(
)( )(jNk
jkjjj xMxb
To find a node’s beliefs: Multiply together all the messages coming in to that node.
![Page 33: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/33.jpg)
Simple BP exampley1
),( 11 yx
),( 21 xx ),( 32 xx
),( 33 yx
x1 x2
y3
x3
),( 21 xx ),( 32 xx
x1 x2 x3
9.1.
1.9.),( 32 xx
6.
4.11yM
9.1.
1.9.),( 21 xx
2.
8.33yM
![Page 34: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/34.jpg)
Simple BP example
9.1.
1.9.),(),( 3221 xxxx
),( 21 xx ),( 32 xx
x1 x2 x3
6.
4.11yM
2.
8.33yM
To find the marginal probability for each variable, you can(a) Marginalize out the other variables of:
(b) Or you can run belief propagation, (BP). BP redistributes the various partial sums, leading to a very efficient calculation.
)()(),(),(),,( 33
311
13221321 xMxMxxxxxxxP yy
![Page 35: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/35.jpg)
Belief, and message updates
jii =
ijNk
jkjji
xi
ji xMxxxM
j \)(ij )(),( )(
j
)(
)( )(jNk
jkjjj xMxb
![Page 36: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/36.jpg)
Optimal solution in a chain or tree:Belief Propagation
• “Do the right thing” Bayesian algorithm.
• For Gaussian random variables over time: Kalman filter.
• For hidden Markov models: forward/backward algorithm (and MAP variant is Viterbi).
![Page 37: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/37.jpg)
No factorization with loops!
y1
x1
y2
x2
y3
x3
),(),(sum
),(),(sum
),(mean
3233
2122
111
3
2
1
xxyx
xxyx
yxx
x
x
xMMSE
31 ),( xx
![Page 38: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/38.jpg)
Justification for running belief propagation in networks with loops
• Experimental results:
– Error-correcting codes
– Vision applications
• Theoretical results:
– For Gaussian processes, means are correct.
– Large neighborhood local maximum for MAP.
– Equivalent to Bethe approx. in statistical physics.
– Tree-weighted reparameterization
Weiss and Freeman, 2000
Yedidia, Freeman, and Weiss, 2000
Freeman and Pasztor, 1999;Frey, 2000
Kschischang and Frey, 1998;McEliece et al., 1998
Weiss and Freeman, 1999
Wainwright, Willsky, Jaakkola, 2001
![Page 39: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/39.jpg)
Outline of MRF section
• Inference in MRF’s.– Iterated conditional modes (ICM)– Gibbs sampling, simulated annealing– Belief propagation– Graph cuts
• Applications of inference in MRF’s.
![Page 40: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/40.jpg)
Graph cuts
• Algorithm: uses node label swaps or expansions as moves in the algorithm to reduce the energy. Swaps many labels at once, not just one at a time, as with ICM.
• Find which pixel labels to swap using min cut/max flow algorithms from network theory.
• Can offer bounds on optimality.• Paper:
Boykov, Veksler, Zabih, IEEE PAMI 2001
![Page 41: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/41.jpg)
Graph cuts
![Page 42: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/42.jpg)
Graph cuts
![Page 43: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/43.jpg)
Source codes (MATLAB)
• Graph Cuts:– http://www.csd.uwo.ca/~olga/code.html
• Belief Propagation:– http://www.di.ens.fr/~mschmidt/Software/UGM.html
![Page 44: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/44.jpg)
Outline of MRF section
• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Belief propagation– Graph cuts
• Applications of inference in MRF’s.
![Page 45: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/45.jpg)
Applications of MRF’s
• Stereo
• Motion estimation
• Labelling shading and reflectance
• Matting
• Texture synthesis
• Many more…
![Page 46: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/46.jpg)
Applications of MRF’s
• Stereo
• Motion estimation
• Labelling shading and reflectance
• Matting
• Texture synthesis
• Many others…
![Page 47: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/47.jpg)
Motion applicationimage patches
image
scene
scene patches
![Page 48: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/48.jpg)
What behavior should we see in a motion algorithm?
• Aperture problem
• Resolution through propagation of information
• Figure/ground discrimination
![Page 49: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/49.jpg)
Aperture Problem
• https://en.wikipedia.org/wiki/Motion_perception#The_aperture_problem
![Page 50: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/50.jpg)
Motion analysis: related work
• Markov network– Luettgen, Karl, Willsky and collaborators.
• Neural network or learning-based– Nowlan & T. J. Senjowski; Sereno.
• Optical flow analysis– Weiss & Adelson; Darrell & Pentland; Ju,
Black & Jepson; Simoncelli; Grzywacz & Yuille; Hildreth; Horn & Schunk; etc.
![Page 51: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/51.jpg)
Motion estimation results (maxima of scene probability distributions displayed)
Initial guesses only show motion at edges.
Iterations 0 and 1
Inference:
Image data
![Page 52: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/52.jpg)
Motion estimation results
Figure/ground still unresolved here.
(maxima of scene probability distributions displayed)
Iterations 2 and 3
![Page 53: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/53.jpg)
Motion estimation results
Final result compares well with vector quantized true (uniform) velocities.
(maxima of scene probability distributions displayed)
Iterations 4 and 5
![Page 54: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/54.jpg)
Vision applications of MRF’s
• Stereo
• Motion estimation
• Labelling shading and reflectance
• Matting
• Texture synthesis
• Many others…
![Page 55: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/55.jpg)
Forming an Image
Surface (Height Map)
Illuminate the surface to get:
The shading image is the interaction of the shapeof the surface, the material, and the illumination
Shading Image
![Page 56: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/56.jpg)
Painting the Surface
Scene
Add a reflectance pattern to the surface. Points inside the squares should reflect less light
Image
![Page 57: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/57.jpg)
Goal
Image Shading Image Reflectance Image
![Page 58: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/58.jpg)
Basic Steps1. Compute the x and y image derivatives2. Classify each derivative as being caused by
either shading or a reflectance change3. Set derivatives with the wrong label to zero. 4. Recover the intrinsic images by finding the least-
squares solution of the derivatives.
Original x derivative image Classify each derivative(White is reflectance)
![Page 59: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/59.jpg)
Learning the Classifiers• Combine multiple classifiers into a strong classifier using
AdaBoost (Freund and Schapire)• Choose weak classifiers greedily similar to (Tieu and Viola
2000)• Train on synthetic images• Assume the light direction is from the right
Shading Training Set Reflectance Change Training Set
![Page 60: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/60.jpg)
Using Both Color and Gray-Scale Information
Results withoutconsidering gray-scale
![Page 61: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/61.jpg)
Some Areas of the Image Are Locally Ambiguous
Input
Shading Reflectance
Is the change here better explained as
or ?
![Page 62: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/62.jpg)
Propagating Information• Can disambiguate areas by propagating
information from reliable areas of the image into ambiguous areas of the image
![Page 63: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/63.jpg)
• Consider relationship between neighboring derivatives
• Use Generalized Belief Propagation to infer labels
Propagating Information
![Page 64: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/64.jpg)
Setting Compatibilities
• Set compatibilities according to image contours– All derivatives along a
contour should have the same label
• Derivatives along an image contour strongly influence each other 0.5 1.0
1
1),( jxx
i
β=
![Page 65: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/65.jpg)
Improvements Using Propagation
Input Image Reflectance ImageWith Propagation
Reflectance ImageWithout Propagation
![Page 66: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/66.jpg)
![Page 67: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/67.jpg)
(More Results)
Input Image Shading Image Reflectance Image
![Page 68: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/68.jpg)
![Page 69: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/69.jpg)
![Page 70: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/70.jpg)
Vision applications of MRF’s
• Stereo
• Motion estimation
• Labelling shading and reflectance
• Matting
• Texture synthesis
• Many others…
![Page 71: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/71.jpg)
GrabCut
http://research.microsoft.com/vision/Cambridge/papers/siggraph04.pdf
![Page 72: Markov Random Fields, Graph Cuts, Belief Propagation Computational Photography Connelly Barnes Slides from Bill Freeman.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56649e165503460f94b00a5c/html5/thumbnails/72.jpg)
Graphical models other than MRF grids
• For a nice on-line tutorial about Bayes nets, see Kevin Murphy’s tutorial in his web page.