Efficient Belief Propagation for Vision Using Linear Constraint...
Transcript of Efficient Belief Propagation for Vision Using Linear Constraint...
![Page 1: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/1.jpg)
Efficient Belief Propagation for Vision Efficient Belief Propagation for Vision Using Linear Constraint NodesUsing Linear Constraint Nodes
Brian Potetz
CVPR 2007
Dept. of Computer Science &Center for the Neural Basis of Cognition
Carnegie Mellon University
![Page 2: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/2.jpg)
• Has been used successfully in many applications– Error-correcting codes, Image super-resolution, stereo,
photometric stereo, etc.
• Slow for large cliques– Each message takes time– Especially slow for continuous variables– Loopy Belief Propagation over continuous variables is
historically limited to pairwise MRFs
• Computes marginals of factorized distributionex:
Belief Propagation
![Page 3: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/3.jpg)
• Has been used successfully in many applications– Error-correcting codes, Image super-resolution, stereo,
photometric stereo, etc.
• Slow for large cliques– Each message takes time– Especially slow for continuous variables– Loopy Belief Propagation over continuous variables is
historically limited to pairwise MRFs
• Computes marginals of factorized distributionex:
Belief Propagation
N = number of variables in clique
M = number of states per variable
![Page 4: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/4.jpg)
Today’s Talk:1) A method to reduce complexity from
to for certain potential functions.2) Application: Higher Order Spatial Priors3) Application: Shape from Shading
Efficient Belief Propagation for Vision Efficient Belief Propagation for Vision Using Linear Constraint NodesUsing Linear Constraint Nodes
![Page 5: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/5.jpg)
Efficient Belief Propagation for Large Cliques
• Each factor sends a message to each associated variable. Requires operations.
Goal of Belief Propagation:Givencompute marginals for each variable.
N = number of variables in the factor. M = number of states per variable.
![Page 6: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/6.jpg)
Efficient Belief Propagation for Large Cliques
• Each factor sends a message to each associated variable. Requires operations.
• Suppose Using a change of variables, each message can be computed in operations
– This is an exact method.
• Method also works if
Goal of Belief Propagation:Givencompute marginals for each variable.
N = number of variables in the factor. M = number of states per variable.
(Linear Constraint Node)
![Page 7: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/7.jpg)
Linear Constraint NodesIf the factor , then I can compute itsmessages in time (instead of time)
(Message from Factor node to Variable node)
![Page 8: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/8.jpg)
Linear Constraint Nodes
Apply a change of variables:
(Jacobian is constant)
(Message from Factor node to Variable node)
If the factor , then I can compute itsmessages in time (instead of time)
![Page 9: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/9.jpg)
Linear Constraint Nodes
Perform each integrand seperately
(Message from Factor node to Variable node)
If the factor , then I can compute itsmessages in time (instead of time)
![Page 10: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/10.jpg)
Linear Constraint Nodes
Perform each integrand seperately
(Message from Factor node to Variable node)
If the factor , then I can compute itsmessages in time (instead of time)
![Page 11: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/11.jpg)
Linear Constraint Nodes
Perform each integrand seperately
(Message from Factor node to Variable node)
If the factor , then I can compute itsmessages in time (instead of time)
![Page 12: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/12.jpg)
Linear Constraint Nodes
Perform each integrand seperately
(Message from Factor node to Variable node)
If the factor , then I can compute itsmessages in time (instead of time)
![Page 13: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/13.jpg)
Linear Constraint Nodes
Perform each integrand seperately
(Message from Factor node to Variable node)
If the factor , then I can compute itsmessages in time (instead of time)
![Page 14: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/14.jpg)
Linear Constraint Nodes
Original Potential Function Approximation using 2 constraint nodes
A sufficient number of linear constraint nodes can approximate any potential function
![Page 15: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/15.jpg)
Linear Constraint Nodes
Original Potential Function Approximation using 4 constraint nodes
A sufficient number of linear constraint nodes can approximate any potential function
![Page 16: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/16.jpg)
Linear Constraint Nodes
Original Potential Function Approximation using 6 constraint nodes
A sufficient number of linear constraint nodes can approximate any potential function
![Page 17: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/17.jpg)
Linear Constraint Nodes
Original Potential Function Approximation using 8 constraint nodes
A sufficient number of linear constraint nodes can approximate any potential function
![Page 18: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/18.jpg)
Hard Linear Constraint Nodes
• Hard constraints allow overcomplete representations.
• Good choice of representation makesLBP more efficientand more effective.
Hard constraints:
![Page 19: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/19.jpg)
Application 1: Inference with Higher-Order Spatial Priors
• Many applications of belief propagation infer images or intrinsic images– image-based rendering, denoising, inpainting– stereo, shape-from-X, reflectance estimation
• These problems are highly ambiguous & require strong spatial priors for success
• Pairwise MRFs cannot capture rich image statistics.
![Page 20: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/20.jpg)
Pairwise MRFs for Denoising
Original ImageImage with Gaussian
additive noise (σ = 20) Pairwise MRF
P. F. Felzenszwalb and D. P. Huttenlocher. Efficient belief propagation for early vision. In CVPR, volume 1, pages 261–268, 2004.
![Page 21: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/21.jpg)
Fields of Experts
• Learn the prior probability of images as a product of student-T distributions on the outputs of K linear filters:
S. Roth and M. J. Black. Fields of experts: A framework for learning image priors. In CVPR, pages 860–867, 2005.
![Page 22: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/22.jpg)
Fields of Experts
• Learn the prior probability of images as a product of student-T distributions on the outputs of K linear filters:
S. Roth and M. J. Black. Fields of experts: A framework for learning image priors. In CVPR, pages 860–867, 2005.
Image patch
![Page 23: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/23.jpg)
Fields of Experts
• Learn the prior probability of images as a product of student-T distributions on the outputs of K linear filters:
S. Roth and M. J. Black. Fields of experts: A framework for learning image priors. In CVPR, pages 860–867, 2005.
Image patch
![Page 24: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/24.jpg)
Fields of Experts• Denoising with FoE is competitive with the current
state-of-the-art using 5x5 filters & gradient descent.• FoE could make a great spatial prior for other
applications (stereo, SFS, etc).• Is efficient belief propagation possible for FoE?
![Page 25: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/25.jpg)
Fields of Experts• Denoising with FoE is competitive with the current
state-of-the-art using 5x5 filters & gradient descent.• FoE could make a great spatial prior for other
applications (stereo, SFS, etc).• Is efficient belief propagation possible for FoE?• Lan, Roth, Huttenlocher, & Black, ECCV 06
– Uses three 2x2 “experts” (clique size = 4)– Uses computational short-cuts particular to denoising.
• Using Linear Contraint Nodes reduces computational complexity from to
X. Lan, S. Roth, D. P. Huttenlocher, and M. J. Black. Efficient belief propagation with learned higher-order markov random fields. In ECCV, pages 269–282, 2006
![Page 26: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/26.jpg)
Denoising ResultsOriginal Image Noisy Image (σ = 20)
Pairwise MRF 2x2 Linear Constraint Nodes
1) Performance is slightly better than Lan et al:
2) Speed is much faster:
3) Design generalizes to other applications.
27.4031.62Ours
27.2930.89Lan et al
26.6630.73Pairwise
σ =20σ =10PSNR:
Lan et al: 16min / iteration
Ours: <2min / iteration
Both require ~30 iterations.
![Page 27: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/27.jpg)
Application 2: Shape from Shading
• Goal: Recover 3D surface shape from a single image– Assume a single, infinitely-distant
light source from a known direction.
– Assume a Lambertian surface with constant albedo.
![Page 28: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/28.jpg)
Shape from Shading• SFS is highly nonlinear and underconstrained• Previous approaches often struggle with local minima.
R. Zhang, P.-S. Tsai, J. E. Cryer, and M. Shah. Shape from shading: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 21(8):690–706, 1999
Zheng & ChellappaLee & KuoGround Truth
![Page 29: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/29.jpg)
Shape from Shading Using LBP
(single pixel)• Two variable nodes per
pixel – representation is overcomplete
• Linear dependencies between variables must be enforced
• Real surfaces have zero curl
Integrability Constraint NodesIntegrability Constraint Nodes
![Page 30: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/30.jpg)
Shape from Shading Using LBP
(single pixel)
Lambertian Constraint NodesLambertian Constraint Nodes• Potential function is a level
set of the Lambertian equation.
• Any BRDF could be used.
![Page 31: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/31.jpg)
Shape from Shading Using LBP
(single pixel)• Encode prior on
and • Here, Laplace distributions
are used.
• Priors on and can be built into
integrability nodes• Smoothness nodes can also
encode convexity priors.
Smoothness NodesSmoothness Nodes
![Page 32: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/32.jpg)
Shape From Shading Results
Good image rerenderings imply that the Lambertian and Integrability constraints were wholly satisfied.
Improved performance can only be achieved through improving the spatial prior model p(Z).
![Page 33: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/33.jpg)
• Any reflectance function can be used– handles matte or specular surfaces
• Any known light arrangement can be used– handles multiple or diffuse lighting
• Handles nondeterministic reflectance functions– handles uncertainty in lighting, reflectance, or albedo
• Handles attached shadows and missing data well.• Computes marginals as well as point estimates.• Can exploit rich spatial priors
– can easily be combined with Fields of Experts• Can be combined with other cues
– Stereo, contours, texture, perspective, etc
Solving SFS using Belief Propagation offers several novel flexibilities:
![Page 34: Efficient Belief Propagation for Vision Using Linear Constraint ...bpotetz/Publications/Potetz_cvpr07...Shape from Shading Using LBP (single pixel) • Two variable nodes per pixel](https://reader035.fdocuments.in/reader035/viewer/2022071412/61095ddf3b8f4710a9483826/html5/thumbnails/34.jpg)
Thank You!
NSF Graduate Research Fellowship to author
NSF IIS-0413211 To Tai Sing Lee
Funded by:
Tai Sing Lee for helpful discussions
Stefan Roth along with X. Lan, D. P. Huttenlocher, M. J. Black for sharing filters & results.
Tom Stepleton
Thanks to: