Variational Bayesian Image Processing on Stochastic Factor Graphs

Click here to load reader

download Variational  Bayesian  Image Processing on Stochastic  Factor Graphs

of 18

  • date post

    31-Jan-2016
  • Category

    Documents

  • view

    41
  • download

    1

Embed Size (px)

description

Variational Bayesian Image Processing on Stochastic Factor Graphs. Xin Li Lane Dept. of CSEE West Virginia University. Outline. Statistical modeling of natural images From old-fashioned local models to newly-proposed nonlocal models Factor graph based image modeling - PowerPoint PPT Presentation

Transcript of Variational Bayesian Image Processing on Stochastic Factor Graphs

  • Variational Bayesian Image Processing on Stochastic Factor GraphsXin LiLane Dept. of CSEEWest Virginia University

  • OutlineStatistical modeling of natural imagesFrom old-fashioned local models to newly-proposed nonlocal models Factor graph based image modelingA powerful framework unifying local and nonlocal approachesEM-based inference on stochastic factor graphsApplications and experimental resultsDenoising, inpainting, interpolation, post-processing, inverse halftoning, deblurring ... ...

    Xin Li - This slide describes the scientific basis of using patches as the units of modeling images: human vision system processes the stimuli through overlapping receptive fields;and engineering concepts of patch: it has appeared in many different forms.

  • Cast Signal/Image Processing Under a Bayesian FrameworkImage restoration (Besag et al.1991)Image denoising (Simoncelli&Adelson1996)Interpolation (Mackay1992) and super-resolution (Schultz& Stevenson1996 )Inverse halftoning (Wong1995)Image segmentation (Bouman&Shapiro1994)x: Unobservable datay: Observation dataImage prior(the focus of this talk)Likelihood(varies from applicationto application)

    Xin Li - This talk is more about likelihood term than image prior (since I am just using BM3D to regularize the reconstruction).

  • Statistical Modeling of Natural Images:the Pursuit of a Good PriorLocal modelsMarkov Random Field (MRF) and its extensions (e.g., 2D Kalman-filtering, Field-of-Expert)Sparsity-based: DCT, wavelets, steerable pyramids, geometric wavelets (edgelets, curvelets, ridgelets, bandelets)Nonlocal modelsBilateral filtering (Tomasi et al. ICCV1998)Texture synthesis (Efros&Leung ICCV1999)Exemplar-based inpainting (Criminisi et al. TIP2004)Nonlocal mean denoising (Buades et al. CVPR2005)Total Least-Square denoising (Hirakawa&Parks TIP2006)Block-matching 3D denoising (Dabov et al. TIP2007)

    Xin Li - Althoug not a common view, it is possible to interpret various image models under a patch-based framework. The main difference between local and nonlocal models lies in the Markovian assumption they made: is it in the domain or the range? Such range-domain duality is the basis for bilateral filtering (arguably the first nonlocal model).

  • Introducing a New Language of Factor Graphs Why Factor Graphs?The most general form of graphical probability models (both MRF and Bayesian networks can be converted to FGs)Widely used in computer science and engineering (forward-backward algorithm, Viterbi algorithm, turbo decoding algorithm, Pearls belief propagation algorithm, Kalman filter1)What is Factor Graph?a bipartite graph that expresses which variables are arguments of which local functionsFactor/function node (solid squares) vs. variable nodes (empty circles)B1B2B7B8B3B4B5B6f1f2f3f4f1f2f3f41,2,43,65,77,8L:F V1Kschischang, F.R.; Frey, B.J.; Loeliger, H.-A., "Factor graphs and the sum-product algorithm," IEEE Transactions on Information Theory,, vol.47, no.2, pp.498-519, Feb 2001

    Xin Li - It might be fair to mention you guys' ICIP2007 work though the targeting application is different (I did not do anything like stochastic approximation here).

  • Variable Nodes=Image PatchesNeuroscience: receptive fields of neighboring cells in human vision system have severe overlapping Engineering: patch has been under the disguise of many different names such as windows in digital filters, blocks in JPEG and the support of wavelet bases Cited from D. Hubel, Eye, Brain and Vision, 1988

    Xin Li - This slide describes the scientific basis of using patches as the units of modeling images: human vision system processes the stimuli through overlapping receptive fields;and engineering concepts of patch: it has appeared in many different forms.

  • Factorization: the Art of Statistical Image ModelingWavelet-based statistical models(geometric proximity defines the neighborhood)Locally linear embedding1(perceptual similarity defines the neighborhood)SPMLDomain-MarkovianRange-Markovian1S.T. Roweis and L.K. Saul, Nonlinear Dimensionality Reduction by Locally Linear Embedding(22 December 2000),Science 290 (5500), 2323.

  • Unification Using Factor Graphsf1f2f3f4B1B2B3B4naive Bayesian(DCT/wavelet-based models)MRF-based B0B1B2B3xB0B1B3B2B0B1B2B3kNN/kmeans clustering(nonlocal image models)

  • A Manifold Interpretation of Nonlocal Image PriorMRNB1BkB0How to maximize the sparsity of a representation?Conventional wisdom: adapt basis to signal (e.g., basis pursuit, matching pursuit)New proposal: adapt signal to basis (by probing its underlying organization principle)

  • Organizing Principle: Latent Variable LP(y|x)xyimage denoisingimage inpaintingimage codingimage halftoningLB11B22B14B13B12B41B31B21B33B32B23B24B34B44B43B42fBfAfCimage deblurringsparsifying transformNature is not economical of structures but organizing principles. - Stanislaw M. Ulam L

  • Maximum-Likelihood Estimation of Graph Structure LPack into3D Array DFor. Trans.CoringInv. Trans.unpack into2D patchesB0BkB1^^^Update theestimate of LUpdate theestimate of xloop over every factor node fjA variational interpretation of such EM-basedinference on FGs is referred to the paperP(y|x)

  • Problem 1: Image DenoisingPSNR(DB) PERFORMANCE COMPARISON AMONG DIFFERENT SCHEMES FOR 12 TEST IMAGES ATw = 100SSIM PERFORMANCE COMPARISON AMONG DIFFERENT SCHEMES FOR 12 TEST IMAGES ATw = 100BM3D(kNN,iter=2)SFG(kmeans,iter=20)worg. 200 400 600 800 1000

  • Problem 2: Image Recoverytop-down: test1, test3, test5top-down: test2, test4, test6DCT FoE EXP BM3D LSP SFGPSNR(dB) performance comparisonSSIM performance comparisonLocal models: DCT, FoE and LSPNonlocal models: EXP, BM3D1 and SFG1Our own extension into image recoveryxy

  • Problem 3: Resolution EnhancementxybicubicNEDI1FG 28.70dB 27.34dB 28.19dB31.76dB 32.36dB 32.63dB34.71dB 34.45dB 37.35dB18.81dB 15.37dB 16.45dB1X. Li and M. Orchard, New edge directed interpolation, IEEE TIP, 2001

    Xin Li - This slides leads to the motivational observation about the limitation of uniform sampling - despite severe aliasing in the last example, the reconstructed image is visually very convincing.

  • Problem 4: Irregular Interpolation29.06dB 31.56dB 34.96dBxyDTKRFG1 28.46dB 31.16dB 36.51dB 17.90dB 18.49dB 29.25dB 26.04dB 24.63dB 29.91dBDT- DelauneyTriangle-based(griddata under MATLAB)

    KR- KernalRegression-based(Takeda et al.IEEE TIP 2007w/o parameteroptimization)1X. Li, Patch-based image interpolation: algorithms and applications, Inter. Workshop on Local and Non-Local Approximation (LNLA)200825% kept

    Xin Li - Nothing new here - just confirm nonuniform sampling could work better with our reconstruction algorithm.

  • Problem 5: Post-processingJPEG-decoded at rate of 0.32bpp(PSNR=32.07dB)

    SFG-enhanced at rate of 0.32bpp(PSNR=33.22dB)

    SPIHT-decoded at rate of 0.20bpp(PSNR=26.18dB)

    SFG-enhanced at rate of 0.20bpp(PSNR=27.33dB)

    Maximum-Likelihood (ML) DecodingMaximum a Posterior (MAP) Decoding

  • Problem 6: Inverse Halftoningwithout nonlocal prior1(PSNR=31.84dB,SSIM=0.8390)with nonlocal prior(PSNR=32.82dB,SSIM=0.8515)1Available from Image Halftoning Toolbox released by UT-Austin Researchers

  • Conclusions and PerspectivesDespite the rich structures in natural images, the underlying organization principle is simple (self-similarityWe have shown how similarity can lead to sparsity in a nonlinear representation of imagesFG only represents one mathematical language for interpreting such principle (multifractal formalism is another)Image processing (low-level vision) could benefit from data clustering (higher-level vision): how does human visual cortex learn to decode the latent variable L through unsupervised learning?

    Reproducible Research: MATLAB codes accompanying this work areavailable at http://www.csee.wvu.edu/~xinl/sfg.html (more will be added)

    Xin Li - I don't know how many in audience know Prof. Kohonen in person. But to me, he is one of my idols - a true pioneer in the field of neural networks.