Sparsity-Based Signal Models and the Sparse K-SVD Algorithm

Slide 1

Sparsity-Based Signal Models and the Sparse K-SVD AlgorithmRon RubinsteinAdvisor: Prof. Michael Elad

October 2010

1Sparse Coding Stage: Global OMPMotivation: adapt the distribution of coefficients to the complexity of the block

Efficiency is achieve by implementing the computations locally only for the affected blockSelect atom+block that produce maximal inner productAdd atom toblock using a single OMP iterationStop when error for entire image reaches desired target

DictionaryImageSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Signal Models

Smooth

Piecewise smooth

Smooth withpoint singularities

?Signal model: a mathematical description of the behavior we expect from a good (uncontaminated ) signal in our systemSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#3

Proof of Equivalence for akLet X and Y be matrices, and let u and v be vectors.Also, assume that vTv=1. Then we can show thatSo, for, we have

Independent of ak!Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Signal Models are EverywhereInterpolation:Bilinear, bicubic: signals are piecewise-smooth Lanczos: signals are band-limitedCompression:PNG: neighboring pixels have similar intensitiesJPEG: small image patches are smoothJPEG2K: images are smooth except for simple singularitiesSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#5AgendaAnalysis and synthesis signal modelsTwo models are better than one

A few words on dictionary designOn dictionaries and choices

Introduction to sparse representationSome background on sparsity and the K-SVD

The sparse dictionary modelIntroducing an efficient and adaptable dictionary!

Why sparse dictionaries are good for youSome uses and applications

Summary and conclusions

Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#6Transform-Based Signal ModelsTransform-based models compute a vector transform of the signal:

Such signal models promote sparsity of (x), in the sense that we expect the coefficients in to decay rapidly.

Wavelet Coefficient Distribution - BarbaraSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#7

Analysis and Synthesis ModelsAnalysis models use a set of linear filters, assumed to produce sparse inner products with the signal:

Synthesis models use a set of atomic signals, assumed to reproduce the input signal via a sparse linear combination:The analysis dictionary contains the linear filters as its rowsThe synthesis dictionary contains the atoms as its columns

DSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#8Analysis versus Synthesis?Obvious question: are the two models equivalent?

Surprising answer:NO!

(Well, except for the invertible case, where we can use D=-1)Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#9Analysis versus Synthesis in the L1 CaseFor the widely-used L1 case,Given any similar-sized D and , we can find large number of signals where the two will substantially differ[Elad, Milanfar & Rubinstein 07]The analysis model is mathematically a subset of the synthesis model, but the mapping is exponentially complexThere are practical cases where one may outperform the other

Geometric structure in the L1 caseSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#10Sparsity measure: how do we quantify the sparsity of the coefficients in x?Current Research on Analysis ModelsL0 (sparsity)Lp normsHuberCauchyDictionary training: can we use computational learning to infer the dictionary from examples?[Black & Roth 05][Rubinstein & Elad 10]Algorithms: how do we efficiently solve the resulting optimization problems?Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#11AgendaAnalysis and synthesis signal modelsTwo models are better than one


Introduction to sparse representationSome background on sparsity and K-SVD



Summary and conclusions

Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#12

Designing Your DictionaryHarmonic Analysis: Analytic dictionaries

Machine Learning:Trained dictionaries[Rubinstein, Bruckstein & Elad 10]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#13Analytic dictionaries arise from a mathematical model of the signals

Analytic DictionariesAtoms have analytic formulationsKnown mathematical properties (e.g. coefficient decay rates)Fast algorithms for computing the transformsLimited expressiveness: all signals behavethe sameSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#14Some Analytic DictionariesFourierGabor

WaveletsCurveletsSmooth signalsSmooth signals

Smooth + point singularitiesSmooth + curve singularities

[Cands & Donoho 99]

Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#15Trained DictionariesDictionary is learned from actual dataFiner adaptation to the target signalsBetter performance in applicationsNon-structured: higher complexity, single scaleTrained dictionaries arise from a set of examples of the signal dataSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Dictionary Training AlgorithmsOlshausen & Field 96MOD (Method of Optimal Directions)[Engan, Aase & Husy 99]

K-SVD [Aharon, Elad & Bruckstein 06]

Generalized PCA[Vidal, Ma & Sastry 05]

Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#17

Dictionary Design: SummaryAnalytic dictionariesLow complexityOptimal for specific classes of signalsNon-adaptive

Trained dictionariesAdaptable to different signal typesBetter results in many applicationsNon-structured Can we have it allSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#18AgendaAnalysis and synthesis signal modelsTwo models are better than one





Summary and conclusionsSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The Sparse Representation ModelWe assumes the existence of a synthesis dictionary whose columns are the atom signals.D

We model natural signals as sparse linear combinations of the dictionary atoms:

We seek exact sparsity of , meaningthat it is assumed to contain mostly zeros.D=xSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Sparse CodingThe equation for is underdetermined:Among all the solutions, we want the sparsest one:

The L0 norm counts the number of non-zeros in a vectorProblem 1 (sparse coding): given a signal x, can we find its representation over D?Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Problem 2 (noisy sparse coding): what if we only have y, a noisy version of x?Noisy Sparse Coding

Use the sparsity assumption to recover and approximate x:

Additive Gaussian noiseSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Sparse Coding AlgorithmsThe sparse coding problem NP-hard in general!Many efficient approximation algorithms:GreedyMP/OMPStOMPLARSCoSAMPIterativeFOCUSSBCRIterated ShrinkageRelaxationBasis PursuitLASSOOthersThresholdingShrinkageRandOMPSuccess bounds available, depending on the sparsity of and the amount of noiseEmpirical performance typically much better than theorySparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Orthogonal Matching Pursuit (OMP)Greedy algorithm selects atoms one at a timeInput: signal y, dictionary DOutput: sparse such that y DInitialize: S = empty atom set residual = y

Stopping condition? = all zeros except on S, non-zeros computed by min-L2NoYesSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#

PSNR = 21.4dB1 atoms / block


Orthogonal Matching Pursuit (OMP)Block size: 8 x 8Overcomplete DCT dictionary, 256 atoms

Overcomplete DCT = extension of DCT with non-integer wave numbers

3 atoms / blockPSNR = 26.6dB





8 atoms / blockPSNR = 32.5dBSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD algorithm:Train an explicit dictionary from examples Which Dictionary?Output.InputSet of ExamplesDTrained DictionaryX[Aharon, Elad & Bruckstein 06]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD Training AlgorithmThe examples are linear combinations of the atomsEach representation uses at most T atomsThe target function to minimize:..XD

Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD: Overview..XDInitialize DSparse CodeDictionary UpdateSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#K-SVD: Sparse Coding Stage

For the jth example

Ordinary Sparse Coding !..XDSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#K-SVD: Dictionary Update Stage

For the kth atom

(Residual)..

,XDSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#K-SVD: Dictionary Update StageWe can do better!

..

But wait! What about sparsity?XDSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#

Only some of the examples use atom dkWhen updating k, only recompute the coefficients for those examplesdkkTSolve with SVDWe want to solve:EkK-SVD: Dictionary Update Stage

~~~Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD: Summary..XDInitialize DictionarySparse CodeUsing OMPDictionary UpdateAtom-by-atom + coeffs.Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD: ApplicationsImage denoising [Elad & Aharon 06]Image inpainting [Raboy 07]Tomography reconstruction [Liao & Sapiro 08]Demosaicing [Mairal, Elad & Sapiro 08]Facial image compression [Bryt & Elad 08]Video denoising [Protter & Elad 09]Image scaling [Zeyde, Elad & Protter 10]Many othersSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Limitations of Explicit DictionariesUnstructured: complex to store and transmitInefficient: applied via explicit matrix multiplicationOver-parameterized: many degrees of freedom require a lot of training examplesDSingle scale: adapted to a specific signal size, inter-scale relations are not expressedUnstructured:Inefficient:Over-parameterized:Single scale:Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#35AgendaAnalysis and synthesis signal modelsTwo models are better than one





Summary and conclusionsSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Parametric DictionariesxAnalytic dictionariesxTrained dictionaries``Parametric DictionariesAdaptivityStructure

`Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#37Some Existing Parametric DictionariesSemi-multiscale K-SVD[Mairal, Sapiro & Elad, 08]

Iterative LS Dictionary Learning Algorithms (ILS-DLA)[Engan, Skretting & Husy, 07]Signature dictionary[Aharon & Elad, 08]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#38Sub-Atomic Particles?

K-SVD dictionary

Could the trained atoms themselves be sparse over some simpler underlying dictionary?Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#39The Sparse Dictionary Model.=.DBABase dictionaryRepresentation matrixX Base dictionarySparse matrixBASparse matrixDouble Sparsity[Rubinstein, Zibulevsky & Elad 10]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#40The Sparse Dictionary Model

BEfficiency: depends mostly on the choice of base dictionary typically an analytic dictionary.Adaptivity: by modifying the representation matrix A..ASparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Complexity of Sparse DictionariesSparse Dictionaries

Signal is d-dimensionalSignal size: n n n = nd = N

Dictionary size: N L, L=O(N)Base dictionary size: N L, separableCardinality of each sparse atom: pAnalyticDictionaries

Explicit DictionariesSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#42

Efficiency of Sparse DictionariesOMP-Cholesky61 secSparse OMP-Cholesky13 secBatch-OMP*4.5 secSparse Batch-OMP*1.2 secExample: patches from pirate encoding time per 1,000 blocks.Signal size:n x n = n2 = N

Dictionary size:N x 2N

Base dictionary:2-D Separable

Atom sparsity: n

Encoding target:n/2 atoms

* [Rubinstein, Zibulevsky & Elad 08]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Assumptions: base dictionary and sparse dictionary are both NxL. Base dictionary is 2-d separable.t

Sparsity-Based Signal Models and the Sparse K-SVD Algorithm

Documents

Transcript of Sparsity-Based Signal Models and the Sparse K-SVD Algorithm