MULTI-MICROPHONE SPEECH DEREVERBERATION USING LIME AND LEAST SQUARES FILTERING

Wavelet for Graphs and its Deployment to Image Processing 1 Michael Elad The Computer Science Department The Technion Israel Institute of technology Haifa 32000, Israel **Joint work with

Idan Ram Israel CohenThe Electrical Engineering department Technion Israel Institute of Technology

The research leading to these results has been received funding from the European union's Seventh Framework Programme (FP/2007-2013) ERC grant Agreement ERC-SPARSE- 320649

12 Patch-Based Processing of ImagesIn the past decade we see more and more researchers suggesting to process a signal or an image with a paradigm of the form:

Break the given image into overlapping (small) patchesOperate on the patches separately or by exploiting inter-relation between themPut back the resulting patches into a result canvas

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 23 Patch-Based Processing of ImagesCommon theme: The image patches are believed to exhibit a highly-structured geometrical form in the embedding space they reside in In the past decade we see more and more researchers suggesting to process a signal or an image with a paradigm of the form: Surprisingly, these methods are very effective, actually leading to todays state-of-the-art in many applications

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 344 Patches Patches Patches Various Ideas:

Non-local-meansKernel regressionSparse representationsLocally-learned dictionariesBM3DStructured sparsityStructural clustering Subspace clustering Gaussian-mixture-modelsNon-local sparse rep.Self-similarityManifold learningWho are the researchers promoting this line of work? Many leading scientists from various places Sorry if I forgot YOU

You& You?

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 4

55 This Talk is About A different way to treat an image using its overlapped patches

Order to form the shortest possible path

Process the PatchesThis ordering induces a very interesting permutation on the image pixelsWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 5

66 Surprisingly, This Talk is Also About In the process, we will find ourselves returning to regular signals, handling them differently

Processing of Non-Conventionally Structured Signals Many signal-processing tools (filters, alg., transforms, ) are tailored for uniformly sampled signals Now we encounter different types of signals: e.g., point-clouds and graphs. Can we extend classical tools to these signals?Our goal: Generalize the wavelet transform to handle this broad family of signalsIn fact, this is how this work started in the first place Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 6Part I GTBWT Generalized Tree-Based Wavelet Transform The Basics7This part is taken from the following two papers: I. Ram, M. Elad, and I. Cohen, Generalized Tree-Based Wavelet Transform, IEEE Trans. Signal Processing, vol. 59, no. 9, pp. 41994209, 2011. I. Ram, M. Elad, and I. Cohen, Redundant Wavelets on Graphs and High Dimensional Data Clouds, IEEE Signal Processing Letters, Vol. 19, No. 5, pp. 291294 , May 2012. Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 7 Problem Formulation8Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 8 Different Ways to Look at This Data9Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 910 Our GoalWavelet Transform Sparse (compact) RepresentationWhy Wavelet? Wavelet for regular piece-wise smooth signals is a highly effective sparsifying transform. However, the signal (vector) f is not necessarily smooth in general.

We would like to imitate this for our data structure.Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 10Diffusion WaveletsR. R. Coifman, and M. Maggioni, 2006.

Multiscale Methods for Data on Graphs and Irregular . SituationsM. Jansen, G. P. Nason, and B. W. Silverman, 2008.

Wavelets on Graph via Spectal Graph TheoryD. K. Hammond, and P. Vandergheynst, and R. Gribonval, 2010.

Multiscale Wavelets on Trees, Graphs and High Supervised LearningM . Gavish, and B. Nadler, and R. R. Coifman, 2010.

Wavelet Shrinkage on Paths for Denoising of Scattered DataD. Heinen and G. Plonka, 2012. Wavelet for Graphs A Wonderful Idea

11I wish we would have thought of it first Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1112 The Main Idea (1) - PermutationPTT-1P-1ProcessingPermutation1D WaveletWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1213 The Main Idea (2) - PermutationIn fact, we propose to perform a different permutation in each resolution level of the multi-scale pyramid:

Naturally, these permutations will be applied reversely in the inverse transform. Thus, the difference between this and the plain 1D wavelet transform applied on f are the additional permutations, thus preserving the transforms linearity and unitarity, while also adapting to the input signal. Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1314 Building the Permutations (1)Lets start with P0 the permutation applied on the incoming signal.

Recall: the wavelet transform is most effective for piecewise regular signals. thus, P0 should be chosen such that P0f is most regular.

So, for example, we can simply permute by sorting the signal f 050100150200250050100150200250050100150200250050100150200250Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1415However: we will be dealing with corrupted signals f (noisy, missing values, ) and thus such a sort operation is impossible.

To our help comes the feature vectors in X, which reflect on the order of the signal values, fk. Recall:

Thus, instead of solving for the optimal permutation that simplifies f, we order the features in X to the shortest path that visits in each point once, in what will be an instance of the Traveling-Salesman-Problem (TSP):

Building the Permutations (2)Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1516 Building the Permutations (3)We handle the TSP task by a greedy (and crude) approximation:

Initialize with an arbitrary index j; Initialize the set of chosen indices to (1)={j}; Repeat k=1:1:N-1 times:Find xi the nearest neighbor to x(k) such that i; Set (k+1)={i}; Result: the set holds the proposed ordering.Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1617 Building the Permutations (4)So far we concentrated on P0 at the finest level of the multi-scale pyramid.

In order to construct P1, P2, , PL-1, the permutations at the other pyramids levels, we use the same method, applied on propagated (reordered, filtered and sub-sampled) feature-vectors through the same wavelet pyramid:

P0LP-Filtering (h) & Sub-sampling P1LP-Filtering (h) & Sub-sampling P2LP-Filtering (h) & Sub-sampling P3LP-Filtering (h) & Sub-sampling Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1718 Why Generalized Tree ? Generalized tree Tree (Haar wavelet)

Our proposed transform: Generalized Tree-Based Wavelet Transform (GTBWT).

We also developed a Redundant version of this transform based on the stationary wavelet transform [Shensa, 1992] [Beylkin, 1992] also related to the A-Trous Wavelet (will not be presented here).Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1819 Treating Graph/Cloud-of-pointsJust to complete the picture, we should demonstrate the (R)GTBWT capabilities on graphs/cloud of points.We took several classical machine learning train + test data for several regression problems, and tested the proposed transform in Cleaning (denoising) the data from additive noise;Filling in missing values (semi-supervised learning); and Detecting anomalies (outliers) in the data.The results are encouraging. We shall present herein one such experiment briefly. SKIP?Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 1920 Treating Graphs: The DataData Set: Relative Location of CT axial axis slices

More details: Overall 53500 such pairs of feature and value, extracted from 74 different patients (43 male and 31 female).

Feature vector of length 384Compute bones and air polar HistogramsLabels: Location in the body [0,180]cm

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2021 Treating Graphs: DenoisingFind for each point its K-NN in feature-space, and compute a weighted average of their labels. . . Original labels+AWGN. . . Noisy labelsDenoising by NLM-like algorithm Apply the RTBWT transform to the point-cloud labels, threshold the values and transform backDenoising by THR with RTBWTWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2122 Treating Graphs: Denoising51015205101520253035noise standard deviationSNR [dB] noisy signalRTBWTNL-meansWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2223 Treating Graphs: Semi-Supervised LearningFind for each missing point its K-NN in feature-space that have a label, and compute a weighted average of their labels. . . Original labels+AWGNFilling-in by NLM-like algorithm Denoising by RTBWTOption: Iterate. . . Noisy and missing labelsDiscard p% of the labels randomly Denoisingby NLM

ProjectionProjection

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2324 Treating Graphs: Semi-Supervised Learning 0.10.20.30.40.50.60.70.80.95101520253035# missing samplesSNR [dB]=20CorruptedNL-meansRTBWT (iter 2)NL-means (iter 2)RTBWT (iter 3)NL-means (iter 3)Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2425 Treating Graphs: Semi-Supervised Learning 0.10.20.30.40.50.60.70.80.95101520253035# missing samplesSNR [dB]=5CorruptedNL-meansRTBWT (iter 2)NL-means (iter 2)RTBWT (iter 3)NL-means (iter 3)Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2527

Now, that the image is organized as a graph (or point- cloud), we can apply the developed transform. The distance measure w(, ) we will be using is Euclidean. It seems that after this conversion, we forget all about spatial proximities. Could an Image Become a Graph? Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2728 Our Transform

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2829 Lets Test It: M-Term ApproximationOriginal Image

Output image

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 2930020004000600080001000010152025303540455055# CoefficientsPSNRGTBWT permutationat varying levelcommon 1D2Ddb4 Lets Test It: M-Term ApproximationFor a 128128 center portion of the image Lenna, we compare the image representation efficiency of the GTBWTA common 1D wavelet transform2D wavelet transform

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 30 Comparison Between Different Wavelets

db1 (Haar)db4db8GTBWTcomparison31Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 31 The Representations Atoms Synthetic Image

Scaling functions Originalimage

waveletswaveletswaveletswaveletswaveletswaveletswaveletswaveletswaveletswaveletswaveletswavelets

32Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 3233

Scaling functions Originalimagewaveletswaveletswaveletswaveletswaveletswaveletswaveletswaveletswaveletswavelets The Representations Atoms Lenna Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 33 Lets Test It: Image Denoising34Noisy imageOutput image

Denoising Algorithm+Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 34 Image Denoising Improvements 35Cycle-spinning: Apply the above scheme several (10) times, with a different GTBWT (different random ordering), and average.

Noisy image GTBWTTHRGTBWT-1Averaging GTBWTTHRGTBWT-1Reconstructed image

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 35P1LP-Filtering (h) & Sub-sampling Image Denoising Improvements Sub-image averaging: A by-product of GTBWT is the propagation of the whole patches. Thus, we get n transform vectors, each for a shifted version of the image and those can be averaged. P0LP-Filtering (h) & Sub-sampling P2LP-Filtering (h) & Sub-sampling P3LP-Filtering (h) & Sub-sampling HP-Filtering (g) & Sub-sampling HP-Filtering (g) & Sub-sampling HP-Filtering (g) & Sub-sampling HP-Filtering (g) & Sub-sampling 36Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 36 Image Denoising Improvements P1LP-Filtering (h) & Sub-sampling P0LP-Filtering (h) & Sub-sampling P2LP-Filtering (h) & Sub-sampling P3LP-Filtering (h) & Sub-sampling HP-Filtering (g) & Sub-sampling HP-Filtering (g) & Sub-sampling HP-Filtering (g) & Sub-sampling HP-Filtering (g) & Sub-sampling Sub-image averaging: A by-product of GTBWT is the propagation of the whole patches. Thus, we get n transform vectors, each for a shifted version of the image and those can be averaged.

Combine these transformed pieces; The center row is the transformed coefficients of f. The other rows are also transform coefficients of d shifted versions of the image.We can reconstruct d versions of the image and average.37Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 37 Image Denoising Improvements 38Restricting the NN: It appears that when searching the nearest-neighbor for the ordering, restriction to near-by area is helpful, both computationally (obviously) and in terms of the output quality.

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 38 Image Denoising Improvements 39Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 39 Image Denoising Results 40We apply the proposed scheme with the Symmlet 8 wavelet to noisy versions of the images Lena and BarbaraFor comparison reasons, we also apply to the two images the K-SVD and BM3D algorithms.

The PSNR results are quite good and competitive. What about run time? /PSNRImageK-SVDBM3DGTBWT10/28.14Lena35.5135.9335.87Barbara34.4434.9834.9425/20.18Lena31.3632.0832.16Barbara29.5730.7230.75Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 403D Transform & threshold3D Transform & threshold Relation to BM3D?41

BM3D Our scheme

GTBWT, and thresholdReorder, In a nut-shell, while BM3D searches for patch neighbors and process them locally, our approach seeks one path through all the patches (each gets its own neighbors as a consequence), and the eventual processing is done globally.Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 4142 What Next?A: Refer to this transform as an abstract sparsification operator and use it in general image processing tasksB: Streep this idea to its bones: keep the patch-reordering, and propose a new way to process imagesWe have a highly effective sparsifying transform for images. It is linear and image adaptiveWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 42Part III Frame Interpreting the GTBWT as a Frame and using it as a Regularizer43This part is documented in the following draft: I. Ram, M. Elad, and I. Cohen, The RTBWT Frame Theory and Use for Images, working draft to be submitted soon.

We rely heavily on Danielyan, Katkovnik, and Eigiazarian, BM3D frames and Variational Image Deblurring, IEEE Trans. on Image Processing, Vol. 21, No. 4, pp. 1715-1728, April 2012.Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 43 Recall Our Core Scheme44Or, put differently, : We refer to GTBWT as a redundant frame, and use a heuristic shrinkage method with it, which aims to approximate the solution of

Synthesis:or Analysis: Noisy imageOutput image

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 4445 Recall: Our Transform (Frame)

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 45 Our Notations46Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 46 What Can We Do With This Frame? 47We could solve various inverse problems of the form:

where: x is the original imagev is an AWGN, and A is a degradation operator of any sort

We could consider the synthesis, the analysis, or their combination:

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 47 Generalized Nash Equilibrium 48* Danielyan, Katkovnik, and Eigiazarian, BM3D frames and Variational Image Deblurring, IEEE Trans. on Image Processing, Vol. 21, No. 4, pp. 1715-1728, April 2012.and solve iteratively*Instead of minimizing the joint analysis/synthesis problem:

break it down into two separate and easy to handle parts:Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 48 Deblurring Results49

Original Blurred RestoredWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 49 Deblurring Results50ImageInput PSNRBM3D-DEB ISNRIDD-BM3D ISNR init. with BM3D-DEBOurs ISNRInit. with BM3D-DEBOurs ISNR 3 iterations with simple initializationLena27.257.957.978.088.20Barbara23.347.807.648.256.21House25.619.329.959.8010.06Cameraman22.238.198.859.198.52Blur PSF =

2=2 Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 50Part IV Patch (Re)-Ordering Lets Simplify Things, Shall We?51This part is based on the paper: I. Ram, M. Elad, and I. Cohen, Image Processing using Smooth Ordering of its Patches, IEEE Transactions on Image Processing, Vol. 22, No. 7, pp. 27642774 , July 2013.Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 51

5252 Returning to the Basics

We extract all (with overlaps) patches of size BB (e.g. B=20)Then we order these patches to form the shortest path, as before

Suppose we start with a clean image What should we expect from this permutation? This reordering induces a permutation on the image pixelsWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 52

5353 Spatial Neighbor Euclidean NeighborSpatial neighbors are not necessarily expected to remain neighbors in the new ordering

010020030040010-410-310-210-1100Sqrt Euclidean DistanceHistogram PeppersBarbaraWhat should we expect? Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 53

54 The Reordered Signal is More RegularWhat should we expect? * Measure of smoothness:

Raster scan: 9.57Hilbert curve: 11.77Sorted (ours): 5.63*

The new path is expected to lead to very smooth (or at least, piece-wise smooth) 1D signal.The ordering is expected to be robust to noise and degradations the underlying signal should still be smooth.Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 545555 Processing the Permuted PixelsAssumptions: After a shortest-path reordering of the patches form a clean image, we expect a highly regular signal.Reordering a corrupted image is likely to lead to a good quality sort as well, due to the robustness brought by the patch-matching.An Idea: Given a corrupted image of the form:where: x is the original imagev is an AWGN, and M is a point-wise degradation operator,Apply this process:

Re-order the pixels to a 1D signalProcess the 1D signal with a simple filterRe-order the pixels back to their location

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 555656 Use the Reordering for Denoising Extract all (with overlaps) patches of size 1010Order these patches as before (TSP)Take the center-row it represents a permutation of the image pixels to a regular functionNoisy with =25 (20.18dB)Smooth/filter the values along this row in a simple wayReconstruction: 32.65dB* This result is obtained with (i) cycle-spinning, (ii) sub-image averaging, (iii) two iterations, (iv) learning the filter , and (v) switched smoothing.*

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 565757 Intuition: Why Should This Work? Ordering based on the noisy pixels Simple smoothingNoisy with =25 (20.18dB)Reconstruction: 32.65dB

True samplesNoisy samplesWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 5758 The Simple Smoothing We DoSimple smoothing works fineoptimize h to minimize the reconstruction MSEOriginal imageNoisy imageCompute the TSP permutationApply the permutation on the pixelsApply the permutation on the pixelsApply a 1D filter h+-We can do better by a training phasebut Naturally, this is done off-line and on other images

Training imageWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 585959 Filtering A Further Improvement

Cluster the patches to smooth and textured sets, and train a filter per each separatelyThe results we show hereafter were obtained by: Cycle-spinningSub-image averaging Two iterationsLearning the filter , and Switched smoothing.Based on patch-STDWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 596060 Denoising Results Using Patch-ReorderingImage/PSNR [dB]10 / 28.1425 / 20.1850 / 14.16LenaK-SVD35.49 31.3627.821st iteration35.3331.5828.542nd iteration35.4131.8129.00BarbaraK-SVD34.4129.5325.401st iteration34.4830.4627.172nd iteration34.4630.5427.45HouseK-SVD36.0032.1228.151st iteration35.5832.4829.372nd iteration35.9432.6529.93Bottom line: (1) This idea works very well;(2) It is especially competitive for high noise levels; and (3) A second iteration almost always pays off. Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 606161 What About Inpainting? Extract all (with overlaps) patches of size 99Order these patches as before distance uses EXISTING pixels onlyTake the center-row it represents a permutation of the image pixels to a regular function0.8 of the pixels are missing Fill the missing values in a simple (cubic interpolation) wayReconstruction: 29.71dB** This result is obtained with (i) cycle-spinning, (ii) sub-image averaging, and (iii) two iterations.

Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 616262 The Rationale0.8 of the pixels are missing Reconstruction: 27.15dBOrderingSimple interpolation

Missing sampleExisting sampleWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 626363 Inpainting Results Examples

Given data 80% missing pixels1st iteration of the proposed alg.DCT and OMP recoveryBi-Cubic interpolation3rd iteration of the proposed alg.Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 636464 Inpainting Results ImageMethodPSNR [dB]LenaBi-Cubic30.25DCT + OMP29.97Proposed (1st iter.)30.25Proposed (2nd iter.)31.80Proposed (3rd iter.)31.96BarbaraBi-Cubic22.88DCT + OMP27.15Proposed (1st iter.)27.56Proposed (2nd iter.)29.34Proposed (3rd iter.)29.71HouseBi-Cubic29.21DCT + OMP29.69Proposed (1st iter.)29.03Proposed (2nd iter.)32.10Proposed (3rd iter.)32.71Reconstruction results from 80% missing pixels using various methods: Bottom line: This idea works very well;It is operating much better than the classic sparse-rep. approach; and Using more iterations always pays off, and substantially so. Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 64Part IV Time to Finish Conclusions and a Bit More65Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 6566 Conclusions66We propose a new wavelet transform for scalar functions defined on graphs or high dimensional data cloudsThe proposed transform extends the classical orthonormal and redundant wavelet transforms We demonstrate the ability of these transforms to efficiently represent and denoise imagesWe also show that the obtained transform can be used as a regularizer in classical image processing Inverse-ProblemsFinally, we show that using the ordering of the patches only, quite effective denoising and inpainting can be obtainedWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 66Sparse Representations and learned dictionaries in the ordered domain?67 What Next ?67Improving the TSP approximation solverDemonstrating the proposed wavelet on more data clouds/graphsWhy TSP? Who says we cannot revisit patches? Replace the TSP ordering by MDS?Replace sub-image averaging with a sparsifying transform Exploiting the known distances?Pixel permutation as regularizer?Lifting scheme for treating clouds? Wavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 676868Thank you for your time

thanks to the organizers of this eventVolkan Cevher and Matthias Seeger

Questions?

Post-Docs Are NeededWavelets for Graphs and Its Deployment to Image ProcessingBy: Michael Elad 68