arXiv:0905.4482v1 [math.NA] 27 May 2009TopicsinCompressedSensingByDeannaNeedellB.S.(UniversityofNevada,Reno)2003M.A.(UniversityofCalifornia,Davis)2005DISSERTATIONSubmittedinpartialsatisfactionoftherequirementsforthedegreeofDOCTOROFPHILOSOPHYinMATHEMATICSintheOFFICEOFGRADUATESTUDIESoftheUNIVERSITYOFCALIFORNIADAVISApproved:RomanVershyninRomanVershynin(Co-Chair)ThomasStrohmerThomasStrohmer(Co-Chair)Jes usDeLoeraJes usDeLoeraCommitteeinCharge2009ic _DeannaNeedell,2009. Allrightsreserved.Idedicatethisdissertationtoallmylovedones,thosewithmetodayandthosewhohavepassedon.iiContents1 Introduction 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 MainIdea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.2 ProblemFormulation. . . . . . . . . . . . . . . . . . . . . . . 31.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.1 ErrorCorrection . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.2 Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.2.3 Radar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 MajorAlgorithmicApproaches 102.1 BasisPursuit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.2 RestrictedIsometryCondition. . . . . . . . . . . . . . . . . . 122.1.3 MainTheorems . . . . . . . . . . . . . . . . . . . . . . . . . . 152.1.4 NumericalResults . . . . . . . . . . . . . . . . . . . . . . . . 182.1.5 LinearProgramming . . . . . . . . . . . . . . . . . . . . . . . 192.1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.2 GreedyMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23iii2.2.1 OrthogonalMatchingPursuit . . . . . . . . . . . . . . . . . . 232.2.2 StagewiseOrthogonalMatchingPursuit . . . . . . . . . . . . 272.2.3 CombinatorialMethods . . . . . . . . . . . . . . . . . . . . . 323 Contributions 363.1 RegularizedOrthogonalMatchingPursuit . . . . . . . . . . . . . . . 363.1.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.1.2 MainTheorems . . . . . . . . . . . . . . . . . . . . . . . . . . 393.1.3 ProofsofTheorems . . . . . . . . . . . . . . . . . . . . . . . . 433.1.4 ImplementationandRuntime . . . . . . . . . . . . . . . . . . 633.1.5 NumericalResults . . . . . . . . . . . . . . . . . . . . . . . . 643.1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 683.2 CompressiveSamplingMatchingPursuit . . . . . . . . . . . . . . . . 703.2.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713.2.2 MainTheorem . . . . . . . . . . . . . . . . . . . . . . . . . . 743.2.3 ProofsofTheorems . . . . . . . . . . . . . . . . . . . . . . . . 773.2.4 ImplementationandRuntime . . . . . . . . . . . . . . . . . . 1053.2.5 NumericalResults . . . . . . . . . . . . . . . . . . . . . . . . 1123.2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1163.3 ReweightedL1-Minimization. . . . . . . . . . . . . . . . . . . . . . . 1203.3.1 MainResults . . . . . . . . . . . . . . . . . . . . . . . . . . . 1233.3.2 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1253.3.3 NumericalResultsandConvergence. . . . . . . . . . . . . . . 1323.4 RandomizedKaczmarz . . . . . . . . . . . . . . . . . . . . . . . . . . 1363.4.1 MainResults . . . . . . . . . . . . . . . . . . . . . . . . . . . 1393.4.2 NumericalExamples . . . . . . . . . . . . . . . . . . . . . . . 142iv4 Summary 145AMatlabCode 149A.1 BasisPursuit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149A.2 OrthogonalMatchingPursuit . . . . . . . . . . . . . . . . . . . . . . 152A.3 RegularizedOrthogonalMatchingPursuit . . . . . . . . . . . . . . . 154A.4 CompressiveSamplingMatchingPursuit . . . . . . . . . . . . . . . . 156A.5 ReweightedL1Minimization. . . . . . . . . . . . . . . . . . . . . . . 158A.6 RandomizedKaczmarz . . . . . . . . . . . . . . . . . . . . . . . . . . 160References 162vTopicsinCompressedSensingAbstractCompressedsensinghasawiderangeof applicationsthatincludeerrorcorrec-tion, imaging, radarandmanymore. Givenasparsesignal inahighdimensionalspace, one wishesto reconstruct that signal accurately and ecientlyfrom a numberoflinearmeasurementsmuchlessthanitsactual dimension. Althoughintheoryitisclearthatthisispossible,thedicultyliesintheconstructionofalgorithmsthatperform the recoveryeciently,as wellas determiningwhichkind of linear measure-mentsallowforthereconstruction. Therehavebeentwodistinctmajorapproachestosparserecoverythateachpresentdierentbenetsandshortcomings. Therst,1-minimization methods such as Basis Pursuit, use a linear optimization problem torecover the signal. This method provides strong guarantees and stability, but relies onLinearProgramming,whosemethodsdonotyethavestrongpolynomiallyboundedruntimes. Thesecondapproachusesgreedymethodsthatcomputethesupportofthe signal iteratively. These methods are usually much faster than Basis Pursuit, butuntilrecentlyhadnotbeenabletoprovidethesameguarantees. Thisgapbetweenthetwoapproacheswasbridgedwhenwedevelopedandanalyzedthegreedyalgo-rithmRegularizedOrthogonal MatchingPursuit(ROMP). ROMPprovidessimilarguaranteestoBasisPursuitaswell asthespeedof agreedyalgorithm. Ourmorerecent algorithm CompressiveSampling Matching Pursuit (CoSaMP) improves upontheseguarantees, andisoptimal ineveryimportantaspect. Recentworkhasalsobeen done on a reweighted version of the 1-minimization method that improves upontheoriginalversionintherecoveryerrorandmeasurementrequirements. Theseal-gorithms are discussedin detail, as well as previous work that servesas a foundationforsparsesignalrecovery.viAcknowledgmentsThereweremanypeoplewhohelpedmeinmygraduatecareer, andIwouldliketotakethisopportunitytothankthem.I amextremelygrateful tomyadviser, Prof. RomanVershynin. I havebeenveryfortunatetohaveanadviserwhogavemethefreedomtoexploremyareaonmyown, andalsotheincredibleguidancetosteermedownadierentpathwhenIreachedadeadend. Romantaughtmehowtothinkaboutmathematicsinawaythatchallengesmeandhelpsmesucceed. Heisanextraordinaryteacher, bothintheclassroomandasanadviser.Myco-adviser, Prof. ThomasStrohmer, hasbeenveryhelpful throughoutmygraduatecareer, andespeciallyinmylast year. Healways madetimefor metotalk about ideas and review papers, despite an overwhelming schedule. His insightfulcommentsandconstructivecriticismshavebeeninvaluabletome.Prof. Jes us DeLoera has been very inspiringto me throughout my time at Davis.His instructionstyle has had a great impact on the way I teach,and I striveto makemyclassroomenvironmentasenjoyableashealwaysmakeshis.Prof. JankoGravnerisanamazingteacher, andchallengeshisstudentstomeethigh standards. The challenges he presented me have given me a respect and fondnessforthesubjectthatstillhelpsinmyresearchtoday.Iwouldalsoliketothankall thestaatUCDavis, andespeciallyCeliaDavis,Perry Gee, and Jessica Potts for all their help and patience. They had a huge impactonmysuccessatDavis,andIamsogratefulforthem.MyfriendsbothinDavisandafarhavehelpedtokeepmesanethroughoutthegraduate process. Our time spent together has been a blessing, and I thank them forviialltheyhavedoneforme. IespeciallythankBlakeforallhisloveandsupport. Hisunwaiveringcondenceinmyabilitieshasmademeastrongerperson.Myveryspecial thankstomyfamily. Myfather, George, hasalwaysprovidedmewithquietencouragementandstrength. Mymother,Debbie,hasshownmehowtoperseverethroughlifechallengesandhasalwaysbeenthereforme. Mysister,Crystal, hashelpedmekeepthejoyinmylife, andgivenmeawonderful nephew,Jack. Mygrandmother andlategrandfather wereaconstant support andhelpedteachmetherewardsofhardwork.Finally, thank youto the NSFfor the nancial support that helped fund parts oftheresearchinthisdissertation.viii1Chapter1Introduction1.1 Overview1.1.1 MainIdeaThephrasecompressedsensing referstotheproblemof realizingasparseinputxusingfewlinearmeasurementsthatpossesssomeincoherenceproperties. Theeldoriginated recently from an unfavorable opinion about the current signal compressionmethodology. The conventionalschemein signal processing,acquiringthe entiresig-nalandthencompressingit,wasquestionedbyDonoho[20]. Indeed,thistechniqueuses tremendous resources toacquireoftenverylargesignals, just tothrowawayinformationduringcompression. Thenaturalquestiontheniswhetherwecancom-bine these twoprocesses,and directlysense the signal or its essential parts usingfewlinear measurements. Recent work in compressedsensinghas answeredthis questioninpositive,andtheeldcontinuestorapidlyproduceencouragingresults.The key objective in compressed sensing (also referred to as sparse signal recoveryorcompressivesampling)istoreconstructasignalaccuratelyandecientlyfroma1.1.Overview 2set of fewnon-adaptivelinear measurements. Signals inthis context arevectors,manyof whichintheapplicationswill representimages. Of course, linearalgebraeasilyshowsthatingeneralitisnotpossibletoreconstructanarbitrarysignalfromanincompleteset of linear measurements. Thus onemust restrict thedomaininwhichthesignalsbelong. Tothisend, weconsider sparse signals, thosewithfewnon-zerocoordinates. Itisnowknownthatmanysignalssuchasreal-worldimagesoraudiosignalsaresparseeitherinthissense,orwithrespecttoadierentbasis.Sincesparsesignalslieinalowerdimensional space,onewouldthinkintuitivelythattheymayberepresentedbyfewlinearmeasurements. Thisisindeedcorrect,butthedicultyisdetermininginwhichlowerdimensional subspacesuchasignallies. Thatis, wemayknowthatthesignalhasfewnon-zerocoordinates, butwedonotknowwhichcoordinatesthoseare. Itisthusclearthatwemaynotreconstructsuchsignals using asimple linear operator, andthat the recoveryrequires moresophisticatedtechniques. Thecompressedsensingeldhasprovidedmanyrecoveryalgorithms,mostwithprovableaswellasempiricalresults.Thereareseveralimportanttraitsthatanoptimalrecoveryalgorithmmustpos-sess. The algorithmneeds tobefast, sothat it canecientlyrecover signals inpractice. Of course, minimal storagerequirementsaswell wouldbeideal. Theal-gorithmshouldprovideuniformguarantees, meaningthatgivenaspecicmethodofacquiringlinearmeasurements, thealgorithmrecoversallsparsesignals(possiblywithhighprobability). Ideally, thealgorithmwouldrequireasfewlinearmeasure-ments as possible. Linear algebra shows us that if a signal has s non-zero coordinates,then recovery is theoretically possible with just 2s measurements. However,recoveryusingonlythispropertywouldrequiresearchingthroughtheexponentiallylargesetof all possible lowerdimensional subspaces,and so in practice is not numericallyfea-1.1.Overview 3sible. Thusinthemorerealisticsetting, wemayneedslightlymoremeasurements.Finally, wewishourideal recoveryalgorithmtobestable. Thismeansthatif thesignal oritsmeasurementsareperturbedslightly, thentherecoveryshouldstill beapproximatelyaccurate. Thisisessential, sinceinpracticeweoftenencounternotonlynoisysignalsor measurements,butalsosignalsthat arenot exactlysparse,butclosetobeingsparse. Forexample,compressiblesignalsarethosewhosecoecientsdecayaccordingtosomepowerlaw. Manysignalsinpracticearecompressible,suchassmoothsignalsorsignalswhosevariationsarebounded.1.1.2 ProblemFormulationSincewewill belookingatthereconstructionof sparsevectors, weneedawaytoquantifythesparsityofavector. Wesaythatad-dimensionalsignalxiss-sparseifithassorfewernon-zerocoordinates:x Rd, |x|0:= [supp(x)[ s d,wherewenotethat ||0isaquasi-norm. For1 p< , wedenoteby ||ptheusualp-norm,|x|p:=_d

i=1[xi[p_1/p,and |x|= max [xi[. Inpractice,signalsareoftenencounteredthat arenotexactlysparse,butwhosecoecientsdecayrapidly. Asmentioned, compressiblesignalsarethosesatisfyingapowerlawdecay:[xi[ Ri(1/q), (1.1.1)1.1.Overview 4wherexisanon-increasingrearrangementof x, Rissomepositiveconstant, and0 < q < 1. Notethatinparticular,sparsesignalsarecompressible.Sparserecoveryalgorithms reconstruct sparsesignals fromasmall set of non-adaptive linear measurements. Each measurement can be viewed as an inner productwiththe signal x Rdandsome vector i Rd(or in Cd)1. If we collect mmeasurementsinthisway, wemaythenconsiderthem dmeasurementmatrixwhosecolumnsarethevectorsi. Wecanthenviewthesparserecoveryproblemasthe recovery of the s-sparse signal x Rdfrom its measurement vector u = x Rm.One of the theoretically simplest ways to recover such a vector from its measurementsu = xistosolvethe0-minimizationproblemminzRd|z|0subjectto z= u. (1.1.2)If xis s-sparse andis one-to-one onall 2s-sparse vectors, thenthe minimizerto (1.1.2) must be the signal x. Indeed, if the minimizer is z, then since x is a feasiblesolution,zmustbes-sparseas well. Sincez= u, z x mustbe inthe kernelof.Butz xis2s-sparse, andsinceisone-to-oneonall suchvectors, wemusthavethatz= x. Thusthis0-minimizationproblemworksperfectlyintheory. However,itisnotnumericallyfeasibleandisNP-Hardingeneral[49,Sec.9.2.2].Fortunately, workincompressedsensing has providedus numericallyfeasiblealternativestothisNP-Hardproblem. Onemajorapproach, BasisPursuit, relaxesthe 0-minimization problem to an 1-minimization problem. Basis Pursuit requires acondition on the measurement matrix stronger than the simple injectivity on sparsevectors, butmanykindsofmatriceshavebeenshowntosatisfythisconditionwith1Althoughsimilarresultsholdformeasurementstakenover thecomplexnumbers,forsimplicityofpresentationweonlyconsiderrealnumbersthroughout.1.1.Overview 5number of measurements m=s logO(1)d. The1-minimizationapproachprovidesuniformguarantees andstability, but relies onmethods inLinear Programming.Since there is yet no known strongly polynomial bound, or more importantly, no linearboundon the runtime of such methods, these approaches are often not optimally fast.TheothermainapproachusesgreedyalgorithmssuchasOrthogonal MatchingPursuit [62], StagewiseOrthogonal MatchingPursuit [23], or IterativeThreshold-ing[27, 3]. Manyof thesemethodscalculatethesupportof thesignal iteratively.Most of theseapproaches workforspecicmeasurement matrices withnumber ofmeasurementsm = O(s log d). Once the support Sof the signal has been calculated,thesignal xcanbereconstructedfromitsmeasurements u=xasx=(S)u,whereSdenotesthemeasurementmatrixrestrictedtothecolumnsindexedbySand denotesthepseudoinverse. Greedyapproachesarefast,bothintheoryandpractice,buthavelackedbothstabilityanduniformguarantees.There has thus existed a gap between the approaches. The 1-minimization meth-ods have provided strong guarantees but have lacked in optimally fast runtimes, whilegreedy algorithms although fast, havelackedin optimal guarantees. We bridged thisgap in the two approaches with our new algorithm Regularized Orthogonal MatchingPursuit(ROMP). ROMPprovidessimilaruniformguaranteesandstabilityresultsasthoseofBasisPursuit,butisaniterativealgorithmsoalsoprovidesthespeedofthegreedyapproach. Ournextalgorithm,CompressiveSamplingMatchingPursuit(CoSaMP)improvesupontheresultsofROMP,andistherstalgorithminsparserecoverytobeprovablyoptimalineveryimportantaspect.1.2.Applications 61.2 ApplicationsThesparserecoveryproblemhasapplicationsinmanyareas,rangingfrommedicineandcodingtheorytoastronomyandgeophysics. Sparsesignalsariseinpracticeinverynatural ways, socompressedsensinglendsitself well tomanysettings. Threedirectapplicationsareerrorcorrection,imaging,andradar.1.2.1 ErrorCorrectionWhensignalsaresentfromonepartytoanother, thesignal isusuallyencodedandgathers errors. Because the errors usually occur in few places, sparse recovery can beused to reconstruct the signal from the corrupted encoded data. This error correctionproblem is a classic problem in coding theory. Coding theory usually assumes the datavalues live in some nite eld, but there are many practical applications for encodingover thecontinuous reals. Indigital communications, for example, onewishes toprotectresultsof onboardcomputationsthatarereal-valued. Thesecomputationsare performed by circuits that experience faults caused by eects of the outside world.Thisandmanyotherexamplesaredicultreal-worldproblemsoferrorcorrection.Theerrorcorrectionproblemisformulatedas follows. Consideram-dimensionalinput vector f Rmthat we wish to transmit reliably to a remote receiver. In codingtheory, thisvectorisreferredtoastheplaintext. Wetransmitthemeasurementsz=Af(orciphertext)whereAisthedmmeasurementmatrix, orthelinearcode. Itisclear thatif thelinear codeAhasfull rank, wecanrecover theinputvectorffromtheciphertextz. Butasisoftenthecaseinpractice,weconsiderthesettingwheretheciphertextzhasbeencorrupted. Wethenwishtoreconstructtheinputsignal ffromthecorruptedmeasurementsz=Af+ ewheree RNisthesparseerrorvector. Torealizethisintheusualcompressedsensingsetting,consider1.2.Applications 7amatrixBwhosekernel istherangeofA. ApplyBtobothsidesoftheequationz= Af+etogetBz= Be. Sety= Bzandtheproblembecomesreconstructingthe sparse vector e from its linear measurements y. Once we have recovered the errorvectore,wehaveaccesstotheactualmeasurementsAfandsinceAisfull rankcanrecovertheinputsignalf.1.2.2 ImagingManyimagesbothinnatureandotherwisearesparsewithrespecttosomebasis.Because of this, many applications in imaging are able to take advantage of the toolsprovidedbyCompressedSensing. Thetypical digital cameratodayrecords everypixel inanimagebeforecompressingthatdataandstoringthecompressedimage.Due to the use of silicon, everyday digital cameras today can operate in the megapixelrange. Anaturalquestionaskswhyweneedtoacquirethisabundanceofdata,justtothrowmostofitawayimmediately. ThisnotionsparkedtheemergingtheoryofCompressiveImaging. In this newframework,the ideais to directlyacquire randomlinearmeasurements of animagewithouttheburdensomestepof capturingeverypixelinitially.Several issues from this of course arise. The rst problem is how to reconstruct theimage from its random linear measurements. The solution to this problem is providedbyCompressedSensing. Thenextissueliesinactuallysamplingtherandomlinearmeasurementswithoutrstacquiringtheentireimage. Researchers[25]areworkingon the construction of a device to do just that. Coined the single-pixel compressivesamplingcamera, thiscameraconsistsof adigital micromirrordevice(DMD), twolenses, asinglephotondetectorandananalog-to-digital(A/D)converter. Therstlens focuses the light onto the DMD. Each mirror on the DMD corresponds to a pixel1.2.Applications 8in the image, and can betilted toward or awayfrom the secondlens. Thisoperationis analogous tocreatinginner products withrandomvectors. This light is thencollectedbythelensandfocusedontothephotondetectorwherethemeasurementiscomputed. Thisoptical computercomputestherandomlinearmeasurementsoftheimageinthiswayandpassesthosetoadigital computerthatreconstructstheimage.Since this camera utilizes only one photon detector, its design is a stark contrast tothe usuallarge photon detectorarray in mostcameras. Thesingle-pixelcompressivesamplingcameraalsooperatesatamuchbroaderrangeofthelightspectrumthantraditional camerasthatusesilicon. Forexample, becausesiliconcannotcaptureawide range of the spectrum, a digital camera to capture infrared images is much morecomplexandcostly.CompressedSensingisalsousedinmedicalimaging,inparticularwithmagneticresonance(MR)imageswhichsampleFouriercoecientsof animage. MRimagesareimplicitlysparseandcanthuscapitalizeonthetheoriesofCompressedSensing.SomeMRimagessuchasangiogramsaresparseintheiractualpixelrepresentation,whereasmorecomplicatedMRimagesaresparsewithrespecttosomeotherbasis,suchasthewaveletFourierbasis. MRimagingingeneralisverytimecostly,asthespeedof datacollectionislimitedbyphysical andphysiological constraints. Thusitisextremelybenecial toreducethenumberof measurementscollectedwithoutsacricing quality of the MR image. Compressed Sensing again provides exactly this,andmanyCompressedSensingalgorithmshavebeenspecicallydesignedwithMRimagesinmind[36,46].1.2.Applications 91.2.3 RadarTherearemanyotherapplicationstocompressedsensing(see[13]), andoneaddi-tional application is Compressive Radar Imaging. A standard radar system transmitssome sort of pulse (for example a linear chirp), and then uses a matched lter to corre-late the signal received with that pulse. The receiver uses a pulse compression systemalong with a high-rate analog to digital (A/D) converter. This conventional approachisnotonlycomplicatedandexpensive, buttheresolutionoftargetsinthisclassicalframeworkislimitedbytheradaruncertaintyprinciple. CompressiveRadarImag-ingtacklestheseproblemsbydiscretizingthetime-frequencyplaneintoagridandconsideringeachpossibletargetsceneasamatrix. Ifthenumberoftargetsissmallenough, thenthegridwill besparselypopulated, andwecanemployCompressedSensingtechniquestorecoverthetargetscene. See[1,39]formoredetails.10Chapter2MajorAlgorithmicApproachesCompressedSensinghasprovidedmanymethodstosolvethesparserecoveryprob-lemandthusitsapplications. Therearetwomajoralgorithmicapproachestothisproblem. Therstreliesonanoptimizationproblemwhichcanbesolvedusinglin-earprogramming,whilethesecondapproachtakesadvantage ofthespeedofgreedyalgorithms. Both approaches have advantages and disadvantages which are discussedthroughoutthischapteralongwithdescriptionsofthealgorithmsthemselves. FirstwediscussBasisPursuit,amethod that utilizesa linearprogram to solvethe sparserecoveryproblem.2.1 BasisPursuitRecall that sparse recovery can be formulated as the generally NP-Hard problem (1.1.2)torecoverasignal x. Donohoandhiscollaboratorsshowed(seee.g. [21])thatforcertainmeasurementmatrices,thishardproblemisequivalenttoitsrelaxation,minzRd|z|1subjectto z= u. (2.1.1)2.1.BasisPursuit 11Cand`es and Tao proved that for measurementmatrices satisfyinga certain quantita-tiveproperty,theprograms(1.1.2)and(2.1.1)areequivalent[6].2.1.1 DescriptionSince the problem(1.1.2) is not numericallyfeasible, it is clear that if one is tosolve the problemeciently, adierent approachis needed. At rst glance, onemayinsteadwishtoconsiderthemeansquareapproach,basedontheminimizationproblem,minzRd|z|2subjectto z= u. (2.1.2)Sincetheminimizerxmustsatisfyx=u=x, theminimizermustbeinthesubspaceKdef=x + ker . Infact, theminimizerxto(2.1.2)isthecontactpointwhere the smallest Euclidean ball centered at the origin meets the subspace K. As isillustrated in Figure 2.1.1, this contact point need not coincide with the actual signalx. Thisisbecausethegeometryof theEuclideanball doesnotlenditself well todetectingsparsity.Wemaythenwishtoconsiderthe1-minimizationproblem(2.1.1). Inthiscase,theminimizerxto(2.1.1)isthecontactpointwherethesmallestoctahedroncen-tered at the origin meets the subspace K. Since x is sparse, it lies in a low-dimensionalcoordinate subspace. Thus the octahedron has a wedge at x (see Figure 2.1.1), whichforcestheminimizerxtocoincidewithxformanysubspacesK.Since the 1-ball works well because of its geometry, one might thinktothenuseanpball forsome012[J0[.Set=J0supp(x). Bythecomparabilitypropertyof thecoordinatesinJ0andsince [[ >12[J0[,thereisafractionofenergyin:|y[|2>15|y[J0|2 17log s|x0|2, (3.1.8)wherethelastinequalityholdsbyLemma3.1.12.Ontheotherhand,wecanapproximateybyy0as|y[|2 |y[y0[|2 +|y0[|2. (3.1.9)Since JandusingLemma3.1.9,wehave|y[y0[|2 2.4|x0|2Furthermore,bydenition(3.1.5)of x0,wehavex0[ = 0. So,byPart1ofProposi-tion3.1.6,|y0[|2 2.03|x0|2.3.1.RegularizedOrthogonalMatchingPursuit 53Usingthelasttwoinequalitiesin(3.1.9),weconcludethat|y[|2 4.43|x0|2.This is acontradictionto(3.1.8) solongas 0.03/log s. This proves Theo-rem3.1.5.ProofofTheorem3.1.2TheproofofTheorem3.1.2parallelsthatofTheorem3.1.1. Webeginbyshowingthat at every iteration of ROMP, either at least 50% of the selected coordinates fromthat iteration are from the support of the actual signal v, or the error bound alreadyholds. ThisdirectlyimpliesTheorem3.1.2.Theorem3.1.13(StableIterationInvariantof ROMP). Let beameasurementmatrixsatisfyingtheRestrictedIsometryConditionwithparameters(4s, )for=0.01/log s. Let xbeanon-zeros-sparsevectorwithmeasurementsu=x + e.ThenatanyiterationofROMP,aftertheregularizationstepwhereIisthecurrentchosenindexset,wehaveJ0 I= and(atleast)oneofthefollowing:(i) [J0 supp(v)[ 12[J0[;(ii) |x[supp(x)\I|2 100log s|e|2.Weshowthat theIterationInvariant implies Theorem3.1.2byexaminingthethreepossiblecases:Case1: (ii)occursatsomeiteration. Werstnotethatsince [I[isnonde-creasing, if(ii)occursatsomeiteration, thenitholdsforall subsequentiterations.To show that this would then imply Theorem 3.1.2, we observe that by the Restricted3.1.RegularizedOrthogonalMatchingPursuit 54IsometryConditionandsince [supp( x)[ [I[ 3s,(1 )| x x|2|e|2 | x x e|2.ThenagainbytheRestrictedIsometryConditionanddenitionof x,| x x e|2 |(x[I) x e|2 (1 +)|x[supp(x)\I|2 +|e|2.Thuswehavethat| x x|2 1 +1 |x[supp(x)\I|2 +21 |e|2.Thus(ii)oftheIterationInvariantwouldimplyTheorem3.1.2.Case2: (i)occursateveryiterationandJ0isalwaysnon-empty. In thiscase, by (i) and the fact that J0is always non-empty, the algorithm identies at leastone elementof the support in every iteration. Thus if the algorithm runs s iterationsor until [I[ 2s, it mustbe that supp(x) I, meaningthat x[supp(x)\I= 0. ThenbytheargumentaboveforCase1,thisimpliesTheorem3.1.2.Case3: (i)occursateachiterationandJ0= forsomeiteration. BythedenitionofJ0,ifJ0= theny = r = 0for thatiteration. Bydenitionof r,thismustmeanthat(x w) + e = 0.This combined with Part 1 of Proposition 3.1.6 below (and its proof, see [55]) appliedwiththesetI= supp(x) Iyields|x w + (e)[I |2 2.03|x w|2.3.1.RegularizedOrthogonalMatchingPursuit 55ThencombininingthiswithPart2ofthesameProposition,wehave|x w|2 1.1|e|2.Sincex[supp(x)\I=(x w)[supp(x)\I, thismeansthattheerrorbound(ii)musthold,sobyCase1thisimpliesTheorem3.1.2.WenowturntotheproofoftheIterationInvariant, Theorem3.1.13. WeproveTheorem3.1.13byinductingoneachiterationofROMP.Wewillshowthatateachiterationthesetof chosenindicesisdisjointfromthecurrentsetI of indices, andthateither(i)or(ii)holds. Clearlyif(ii)heldinapreviousiteration,itwouldholdin all future iterations. Thuswemayassumethat (ii) has not yetheld. Since (i) hasheldateachpreviousiteration,wemusthave[I[ 2s. (3.1.10)Consideran iteration ofROMP, andlet r ,= 0be the residualat the start ofthatiteration. LetJ0andJbethesetsfoundbyROMPinthisiteration. Asin[55],weconsiderthesubspaceH:= range(supp(v)I)anditscomplementarysubspacesF:= range(I), E0:= range(supp(v)\I).Part 3 of Proposition 3.1.6 states that the subspaces Fand E0are nearly orthogonal.3.1.RegularizedOrthogonalMatchingPursuit 56For thisreasonweconsiderthesubspace:E:= F H.Firstwewritetheresidualrintermsofprojectionsontothesesubspaces.Lemma 3.1.14 (Residual).Here and onward, denote by PL the orthogonal projectioninRmontoalinearsubspaceL. Thentheresidual rhasthefollowingform:r = PEx +PFe.Proof. By denition of the residual r in the ROMP algorithm, r = PFu = PF(x+e). Tocompletetheproof weneedthat PFx=PEx. This follows fromtheorthogonal decompositionH= F+Eandthefactthatx H.Nextweexaminethemissingportionofthesignalaswellasitsmeasurements:x0:= x[supp(x)\I, u0:= x0 E0. (3.1.11)InthenexttwolemmasweshowthatthesubspacesEandE0areindeedclose.Lemma3.1.15(Approximation of the residual). Letrbetheresidualvectorandu0asin(3.1.11). Then|u0r|2 2.2|u0|2 +|e|2.Proof. Sincex x0hassupportinI, wehavex u0=(x x0) F. ThenbyLemma3.1.7,r = PEx +PFe = PEu0 +PFe. Therefore,|x0r|2= |x0 PEx0 PFe|2 |PFx0|2 +|e|2.3.1.RegularizedOrthogonalMatchingPursuit 57Notethat by(3.1.10), theunionof thesets I andsupp(x) I hascardinalitynogreaterthan3s. ThusbyPart3ofProposition3.1.6,wehave|PFu0|2 +|e|2= |PFPE0u0|2 +|e|2 2.2|u0|2 +|e|2.Lemma3.1.16(Approximationof theobservation). Let y0=u0andy=r.ThenforanysetT 1, . . . , dwith [T[ 3s,|(y0y)[T|2 2.4|x0|2 + (1 +)|e|2.Proof. ByLemma3.1.15andtheRestrictedIsometryConditionwehave|u0r|2 2.2|x0|2 +|e|2 2.2(1 +)|x0|2 +|e|2 2.3|x0|2 +|e|2.ThenbyPart2ofProposition3.1.6wehavethedesiredresult,|(y0y)[T|2 (1 +)|u0r|2.Theresultofthetheoremrequiresustoshowthatwecorrectlygainaportion ofthe supportofthesignalx. Tothisend,werstshowthat ROMPcorrectlychoosesaportionoftheenergy. Theregularizationstepwillthenimplythatthesupportisalsoselectedcorrectly. WethusnextshowthattheenergyofywhenrestrictedtothesetsJandJ0issucientlylarge.3.1.RegularizedOrthogonalMatchingPursuit 58Lemma3.1.17(Localizingtheenergy). Let ybetheobservationvectorandx0beasin(3.1.11). Then |y[J|2 0.8|x0|2(1 +)|e|2.Proof. LetS= supp(x) Ibethemissingsupport. Since [S[ s,bydenitionof Jinthealgorithm,wehave|y[J|2 |y[S|2.ByLemma3.1.16,|y[S|2 |y0[S|22.4|x0|2(1 +)|e|2.Sincex0[S= x0,Part1ofProposition3.1.6implies|y0[S|2 (1 2.03)|x0|2.Thesethreeinequalitiesyield|y[J|2 (1 2.03)|x0|22.4|x0|2(1 +)|e|2 0.8|x0|2(1 +)|e|2.Thiscompletestheproof.Lemma3.1.18(Regularizing the energy). Againletybetheobservationvector andx0beasin(3.1.11). Then|y[J0|2 14log s|x0|2 |e|22log s.Proof. ByLemma3.1.11appliedtothevectory[J,wehave|y[J0|2 12.5log s|y[J|2.3.1.RegularizedOrthogonalMatchingPursuit 59AlongwithLemma3.1.17thisimpliestheclaim.We nowconcludethe proof ofTheorem3.1.13. Theclaim that J0I= followsbythesameargumentsasin[55].Itremainstoshowitslastclaim, thateither(i)or(ii)holds. Suppose(i)inthetheoremfails. Thatis,suppose [J0 supp(x)[ 12[J0[.Set= J0supp(x). BythedenitionofJ0inthealgorithmandsince [[ >12[J0[,wehavebyLemma3.1.18,|y[|2>15|y[J0|2 145 log s|x0|2 |e|225 log s. (3.1.12)Next,wealsohave|y[|2 |y[y0[|2 +|y0[|2. (3.1.13)Since Jand [J[ s,byLemma3.1.16wehave|y[y0[|2 2.4|x0|2 + (1 +)|e|2.Bythedenitionof x0in(3.1.11), itmust bethatx0[=0. Thus byPart 1ofProposition3.1.6,|y0[|2 2.03|x0|2.3.1.RegularizedOrthogonalMatchingPursuit 60Usingthepreviousinequalitiesalongwith(3.1.13),wededucethat|y[|2 4.43|x0|2 + (1 +)|e|2.Thisisacontradictionto(3.1.12)whenever 0.02log s|e|2|x0|2.If this is true, then indeed (i) in the theorem must hold. If it is not true, then by thechoiceof,thisimpliesthat|x0|2 100|e|2_log s.ThisprovesTheorem3.1.13.ProofofCorollary3.1.3Proof. Werstpartition xsothat u = x2s +(x x2s) +e. Thensincesatisesthe Restricted Isometry Condition with parameters (8s, ), by Theorem 3.1.2 and thetriangleinequality,|x2s x|2 104_log 2s(|(x x2s)|2 +|e|2), (3.1.14)The following lemma as in [32] relates the 2-norm of a vectors tail to its 1-norm. Anapplicationofthislemmacombinedwith(3.1.14)willproveCorollary3.1.3.Lemma3.1.19(Comparingthenorms). Letv Rd,andletvTbethevectorofthe3.1.RegularizedOrthogonalMatchingPursuit 61Tlargestcoordinatesinabsolutevaluefromv. Then|v vT|2 |v|12T.Proof. Bylinearity, wemayassume |v|1=d. SincevTconsists of thelargest Tcoordinatesofvinabsolutevalue,wemusthavethat |v vT|2 d T. (Thisisbecause the term |v vT|2is greatest when the vector vhas constant entries.) ThenbytheAM-GMinequality,|v vT|2T d TT (d T+T)/2 = d/2 = |v|1/2.Thiscompletestheproof.ByLemma29of[32],wehave|(x x2s)|2 (1 +)_|x x2s|2 + |x x2s|1s_.ApplyingLemma3.1.19tothevectorv= x xswethenhave|(x x2s)|2 1.5(1 +)|x xs|1s.Combinedwith(3.1.14),thisprovesthecorollary.ProofofCorollary3.1.4Oftenonewishestondasparseapproximationtoasignal. Wenowshowthatbysimplytruncatingthereconstructedvector, weobtaina2s-sparsevectorveryclose3.1.RegularizedOrthogonalMatchingPursuit 62totheoriginalsignal.Proof. LetxS:= x2sand xT:= x2s,andletSandTdenotethesupportsofxSand xTrespectively. By Corollary 3.1.3, it suces to show that |xS xT|2 3|xS x|2.Applyingthetriangleinequality,wehave|xS xT|2 |(xS xT)[T|2 +|xS[S\T|2=: a +b.Wethenhavea = |(xS xT)[T|2= |(xS x)[T|2 |xS x|2andb | x[S\T|2 +|(xS x)[S\T|2.Since [S[ = [T[,wehave [ST[ = [TS[. BythedenitionofT,everycoordinateof x in Tis greater than or equal to every coordinate of x in Tcin absolute value. Thuswehave,| x[S\T|2 | x[T\S|2= |(xS x)[T\S|2.Thusb 2|xS x|2,andsoa +b 3|xS x|2.Thiscompletestheproof.Remark. Corollary3.1.4combinedwithCorollary3.1.3and (3.1.2) impliesthatwe3.1.RegularizedOrthogonalMatchingPursuit 63canalsoestimateaboundonthewholesignalv:|x x2s|2 C_log 2s_|e|2 + |x xs|1s_.3.1.4 ImplementationandRuntimeThe Identication step of ROMP, i.e. selection of the subset J, can be done by sortingthe coordinates of yin the nonincreasingorder and selecting s biggest. Many sortingalgorithmssuchasMergesortorHeapsortproviderunningtimesofO(d log d).TheRegularizationstepof ROMP, i.e. selectingJ0 J, canbedonefast byobserving that J0 is an interval in the decreasing rearrangement of coecients. More-over, theanalysisofthealgorithmshowsthatinsteadofsearchingoverallintervalsJ0, it suces to look for J0 among O(log s) consecutive intervals with endpoints wherethe magnitudeofcoecientsdecreasesbyafactor of 2. (theseare thesetsAkin theproof of Lemma 3.1.11). Therefore, the Regularization step can be done in time O(s).In addition to these costs, the k-th iteration step of ROMP involves multiplicationofthedmmatrixbyavector, andsolvingtheleastsquaresproblemwiththed[I[ matrix I, where [I[ 2s. For unstructured matrices, these tasks can be donein time dm and O(s2m) respectively[2]. Since the submatrix of when restricted tothe indexsetIis nearan isometry,usingan iterativemethodsuchasthe ConjugateGradient Method allows us to solve the least squares method in a constant number ofiterations (up to a specic accuracy)[2, Sec.7.4]. Usingsuch a method then reducesthe time of solvingthe least squares problemtojust O(sm). Thus inthe caseswhereROMPterminatesafteraxednumberofiterations, thetotal timetosolve3.1.RegularizedOrthogonalMatchingPursuit 64all requiredleastsquaresproblemswouldbejustO(sm). Forstructuredmatrices,suchaspartial Fourier, thesetimescanbeimprovedevenmoreusingfastmultiplytechniques.Inothercases, however, ROMPmayneedmorethanaconstantnumberofiter-ationsbeforeterminating, saythefullO(s)iterations. Inthiscase, itmaybemoreecientto maintain the QR factorization of Iand use the Modied Gram-Schmidtalgorithm. With thismethod,solvingall theleastsquaresproblemstakestotal timejustO(s2m). However, storingtheQRfactorizationisquitecostly, soinsituationswhere storage is limited it may be best to use the iterative methods mentioned above.ROMPterminatesinatmost2siterations. Therefore,forunstructuredmatricesusing the methods mentioned above and in the interesting regime m log d, the totalrunningtimeofROMPisO(dNn). ThisisthesameboundasforOMP[62].3.1.5 NumericalResultsNoiselessNumericalStudiesThissectiondescribesourexperimentsthatillustratethesignal recoverypowerofROMP,asshownin[55]. SeeSectionA.3fortheMatlabcodeusedinthesestudies.We experimentallyexamine howmanymeasurements mare necessarytorecovervariouskindsofs-sparsesignalsinRdusingROMP. Wealsodemonstratethatthenumberof iterationsROMPneedstorecoverasparsesignal isinpracticeatmostlinearthesparsity.Firstwedescribethesetupofourexperiments. Formanyvaluesoftheambientdimensiond, thenumber of measurements m, andthe sparsitys, we reconstructrandomsignalsusingROMP.For eachsetofvalues,wegenerateanmdGaussianmeasurement matrixandthenperform500independent trials. The results we3.1.RegularizedOrthogonalMatchingPursuit 65obtainedusingBernoulli measurementmatriceswereverysimilar. Inagiventrial,we generate an s-sparse signal x in one of two ways. In either case, we rst select thesupportofthesignal bychoosingscomponentsuniformlyatrandom(independentfrom the measurement matrix ). In the cases where we wish to generate at signals,we then set these components to one. Our work as well as the analysis of Gilbert andTropp[62]showthatthisisachallengingcaseforROMP(andOMP).Inthecaseswherewewishtogeneratesparsecompressiblesignals, wesettheithcomponentofthe support to plus or minus i1/pfor a specied value of 0 < p < 1. We then executeROMPwiththemeasurementvectoru = x.Figure 3.1.1 depicts the percentage (from the 500 trials) of sparse at signals thatwerereconstructedexactly. Thisplotwasgeneratedwithd=256forvariouslevelsof sparsitys. Thehorizontal axisrepresentsthenumberof measurements m, andthevertical axisrepresentstheexactrecoverypercentage. WealsoperformedthissametestforsparsecompressiblesignalsandfoundtheresultsverysimilartothoseinFigure3.1.1. OurresultsshowthatperformanceofROMPisverysimilartothatofOMPwhichcanbefoundin[62].Figure3.1.2depictsaplotofthevaluesformandsatwhich99%ofsparseatsignalsarerecoveredexactly. Thisplotwasgeneratedwithd = 256. Thehorizontalaxisrepresents thenumber of measurements m, andthevertical axisthesparsitylevels.Theorem3.1.1guaranteesthatROMPrunswithatmostO(s)iterations. Fig-ure3.1.3depictsthenumberof iterationsexecutedbyROMPford=10, 000andm = 200. ROMPwasexecutedunderthe samesettingasdescribedabovefor sparseat signals as well as sparsecompressible signals for various values of p, andthenumberofiterationsineachscenariowasaveragedoverthe500trials. Theseaver-3.1.RegularizedOrthogonalMatchingPursuit 66ageswereplottedagainstthesparsityof thesignal. Astheplotillustrates, only2iterations wereneededfor at signals evenfor sparsitys as high as 40. The plot alsodemonstratesthatthenumberofiterationsneededforsparsecompressibleishigherthanthenumberneededforsparseatsignals, asonewouldexpect. Theplotsug-geststhatforsmallervaluesofp(meaningsignalsthatdecaymorerapidly)ROMPneedsmoreiterations. However itshowsthateveninthecaseof p=0.5, only6iterationsareneededevenforsparsitysashighas20.0 50 100 150 200 2500102030405060708090100Number of Measurements (N)Percentage recoveredPercentage of input signals recovered exactly (d=256) (Gaussian) n=4n=12n=20n=28n=36Figure3.1.1: ThepercentageofsparseatsignalsexactlyrecoveredbyROMPasafunctionofthenumberofmeasurementsindimensiond=256forvariouslevelsofsparsity.NoisyNumericalStudiesThis section describes our numerical experiments that illustrate the stability of ROMPas shownin[54]. Westudytherecoveryerror usingROMPfor both perturbedmea-surements andsignals. Theempirical recoveryerrorisactuallymuchbetter than3.1.RegularizedOrthogonalMatchingPursuit 670 50 100 150 200 250051015202530354045Measurements NSparsity Level99% Recovery Trend, d=256Figure 3.1.2: The 99% recoverylimit as a function of the sparsityand the number ofmeasurementsforsparseatsignals.thatgiveninthetheorems.First we describe the setup to our experimental studies. We run ROMP on variousvalues of the ambientdimensiond, the numberof measurementsm,and the sparsitylevel s, andattempttoreconstructrandomsignals. Foreachsetofparameters, weperform500trials. Initially, wegenerateanmdGaussianmeasurement matrix. Foreachtrial, independent of thematrix, wegenerateans-sparsesignal xbychoosings componentsuniformlyatrandom andsettingthemtoone. Inthecaseofperturbedsignals, weaddtothesignal ad-dimensional errorvectorwithGaussianentries. In the case of perturbed measurements, we add an m-dimensional error vectorwithGaussianentriestothemeasurementvectorx. WethenexecuteROMPwiththemeasurementvectoru = xoru + eintheperturbedmeasurementcase. AfterROMPterminates, weoutput thereconstructedvector xobtainedfromtheleastsquarescalculationandcalculateitsdistancefromtheoriginalsignal.3.1.RegularizedOrthogonalMatchingPursuit 680 5 10 15 200123456Sparsity Level (n)Number of IterationsNumber of Iterations needed to reconstruct (d=10,000, N=200) (Gaussian) p=0.5p=0.7p=0.9Flat SignalFigure 3.1.3: The number of iterations executedbyROMPas afunctionof thesparsityindimensiond = 10, 000with200measurements.Figure3.1.4depictstherecoveryerror |x x|2whenROMPwasrunwithper-turbedmeasurements. Thisplotwasgeneratedwithd=256forvariouslevelsofsparsitys. Thehorizontal axisrepresentsthenumberofmeasurementsm, andtheverticalaxisrepresentstheaveragenormalizedrecoveryerror. Figure3.1.4conrmstheresultsofTheorem3.1.2, whilealsosuggestingtheboundmaybeimprovedbyremovingthelog sfactor.Figure 3.1.5 depictsthe normalized recoveryerror when the signal was perturbedbyaGaussianvector. Thegureconrmstheresultsof Corollary3.1.3whilealsosuggestingagainthatthelogarithmicfactorinthecorollaryisunnecessary.3.1.6 SummaryThereareseveral critical propertiesthatanideal algorithmincompressedsensingshould possess. One such property is stability, guaranteeing that under small pertur-3.1.RegularizedOrthogonalMatchingPursuit 6960 80 100 120 140 160 180 200 220 240 26000.511.5Measurements mError to Noise RatioNormalized Recovery Error from ROMP on Perturbed Measurements , d=256 s=4s=12s=20s=28s=36Figure 3.1.4: The error to noiseratio xx2e2as a functionof the numberofmeasure-mentsmindimensiond = 256forvariouslevelsofsparsitys.bationsoftheinputs, thealgorithmstill performsapproximatelycorrect. Secondly,thealgorithmneedstoprovideuniformguarantees, meaningthatwithhighproba-bilitythealgorithmworkscorrectlyforall inputs. Finally, tobeideal inpractice,thealgorithmwouldneedtohaveafastruntime. The1-minimizationapproachtocompressedsensingisstableandprovidesuniformguarantees, butsinceitreliesonthe use of Linear Programming, lacks a strong bound on its runtime. The greedy ap-proach is quite fast both in theory and in practice, but had lackedboth stability anduniformguarantees. Weanalyzedtherestrictedisometrypropertyinauniquewayandfoundconsequencesthatcouldbeusedinagreedyfashion. OurbreakthroughalgorithmROMPisthersttoprovideallthesebenets(stability,uniformguaran-tees,andspeed),andessentiallybridgesthegapbetweenthetwomajorapproachesincompressedsensing.3.2.CompressiveSamplingMatchingPursuit 700 50 100 150 200 250 30000.10.20.30.40.50.60.70.8Measurements mError to Noise RatioNormalized Recovery Error from ROMP on Perturbed Signals , d=256 s=4s=12s=20s=28s=36Figure 3.1.5: The error to noise ratio xx2s

2xxs1/susing a perturbed signal, as a functionof the numberof measurementsm in dimensiond = 256 for various levelsof sparsitys.3.2 CompressiveSamplingMatchingPursuitRegularizedOrthogonalMatchingPursuitbridgedacriticalgapbetweenthemajorapproaches in compressedsensing. It provided the speed of the greedy approach andthestrongguaranteesoftheconvexoptimizationapproach. Althoughitscontribu-tions were signicant, it still left room for improvement. The requirements imposed byROMP on the restricted isometry condition were slightly stronger than those imposedbytheconvexoptimizationapproach. Thistheninturnweakenedtheerrorboundsprovided by ROMP in the case of noisy signals and measurements. These issues wereresolvedbyour algorithmCompressive SamplingMatchingPursuit (CoSaMP). Asimilaralgorithm,SubspaceMatchingPursuitwasalsodevelopedaroundthistime,whichprovidessimilarbenetstothoseofCoSaMP.See[14]fordetails.3.2.CompressiveSamplingMatchingPursuit 713.2.1 DescriptionOne of the key dierences between ROMP and OMP is that at each iteration ROMPselectsmorethanonecoordinatetobeinthesupportset. Becauseofthis, ROMPisabletomakemistakesinthesupportset, whilestill correctlyreconstructingtheoriginal signal. This is accomplishedbecause we boundthe number of incorrectchoices the algorithm can make. Once the algorithm chooses an incorrect coordinate,however, thereisnowayforittoberemovedfromthesupportset. Analternativeapproachwouldbe toallowthealgorithmtochoose incorrectlyas well as xitsmistakesinlateriterations. Inthiscase,ateachiterationweselectaslightlylargersupportset, reconstructthesignal usingthatsupport, andusethatestimationtocalculatetheresidual.Tropp andNeedell developed a newvariant of OMP, Compressive SamplingMatchingPursuit (CoSaMP) [53, 52]. This newalgorithmhas thesameuniformguaranteesasROMP, butdoesnotimposethelogarithmictermfortheRestrictedIsometryPropertyorintheerrorbounds. SincethesamplingoperatorsatisestheRestrictedIsometryProperty, everysentriesofthesignal proxyy=xarecloseintheEuclideannormtothescorrespondingentriesofthesignalx. ThusasinROMP, thealgorithmrstselects thelargestcoordinatesof thesignal proxyyand adds thesecoordinates tothe runningsupport set. Nexthowever,thealgorithmperforms a least squares step to get an estimate b of the signal, and prunes the signaltomakeitsparse. Thealgorithmsmajorstepsaredescribedasfollows:1. Identication. Thealgorithmformsaproxyoftheresidualfromthecurrentsamplesandlocatesthelargestcomponentsoftheproxy.2. SupportMerger. Thesetofnewlyidentiedcomponentsisunitedwiththe3.2.CompressiveSamplingMatchingPursuit 72setofcomponentsthatappearinthecurrentapproximation.3. Estimation. The algorithm solves a least-squares problem to approximate thetargetsignalonthemergedsetofcomponents.4. Pruning. Thealgorithmproducesanewapproximationbyretainingonlythelargestentriesinthisleast-squaressignalapproximation.5. SampleUpdate. Finally, thesamplesareupdatedsothattheyreecttheresidual,thepartofthesignalthathasnotbeenapproximated.Thesestepsare repeateduntilthe haltingcriterionis triggered. Initially,weconcen-trate onmethodsthatuseaxednumberofiterations. Section3.2.4discussessomeothersimplestoppingrulesthatmayalsobeuseful inpractice. Usingtheseideas,thepseudo-codeforCoSaMPcanthusbedescribedasfollows.3.2.CompressiveSamplingMatchingPursuit 73CompressiveMatchingPursuit(CoSaMP)[53]Input: Measurementmatrix,measurementvectoru = x,sparsitylevelsOutput: s-sparsereconstructedvector x = aProcedure:InitializeSet a0= 0, v= u, k= 0. Repeat the following steps and incrementkuntilthehaltingcriterionistrue.Signal ProxySety=v, =suppy2sandmergethesupports: T = suppak1.Signal EstimationUsingleast-squares,setb[T= Tuandb[Tc= 0.PruneToobtainthenextapproximation,setak= bs.SampleUpdateUpdatethecurrentsamples: v = u ak.Thereareafewmajorconcepts of whichthealgorithmCoSaMPtakesadvan-tage. Unlikesomeothergreedyalgorithms, CoSaMPselects manycomponentsateachiteration. Thisideacanbefoundintheoretical workongreedyalgorithmsbyTemlyakov as well as some early work of Gilbert, Muthukrishnan,Strauss and Tropp[34],[63]. It is also the key idea of recent work on the Fourier sampling algorithm [35].TheROMPandStOMPalgorithmsalsoincorporatethisnotion[55],[23].Theapplicationof theRestrictedIsometryPropertytocomparethenorms ofvectorsundertheactionofthesamplingoperatoranditsadjointisalsokeyinthisalgorithmanditsanalysis. TheRestrictedIsometryPropertyisduetoCand`esandTao[9]. Theapplicationofthepropertytogreedyalgorithmsisrelativelynew,andappearsin[32]and[55].Another key idea present in the algorithm is the pruning step to maintain sparsity3.2.CompressiveSamplingMatchingPursuit 74of theapproximation. Thisalsohassignicantramicationsinotherpartsof theanalysis and the running time. Since the Restricted Isometry Property only holds forsparsevectors,itisvitalintheanalysisthattheapproximationremainsparse. Thisideaalsoappearsin[32].Our analysis focuses on mixed-norm error bounds. This ideaappears in the workof Cand`es,Romberg, Tao [5] as well as [32] and [10]. In our analysis, we focus on thefactthatiftheerrorislarge,thealgorithmmustmakeprogress. ThisideaappearsinworkbyGilbertandStrauss,forexample[32].The L1-minimizationmethodandthe ROMPalgorithmprovide the strongestknownguarantees of sparse recovery. These guarantees areuniforminthat oncethe sampling operator satises the Restricted Isometry Property, both methods workcorrectlyfor all sparse signals. L1-minimizationis basedonlinear programming,whichprovides onlyapolynomial runtime. Greedyalgorithms suchas OMPandROMPonthe other hand, are muchfaster bothintheoryandempirically. OuralgorithmCoSaMPprovidesbothuniformguaranteesaswell asfastruntime, whileimprovingupontheerrorboundsandRestrictedIsometryrequirementsof ROMP.Wedescribetheseresultsnextaswestatethemaintheorems.3.2.2 MainTheoremNext we statethe maintheoremwhichguarantees exact reconstructionof sparsesignals and approximate reconstruction of arbitrary signals. The proof of the theoremispresentedinSection3.2.3.Theorem3.2.1(CoSaMP[53]). Supposethatisanm dsamplingmatrixwithrestrictedisometryconstant 2sc, as in (2.1.4). Let u=x+ e be avectorof samples of anarbitrarysignal, contaminatedwitharbitrarynoise. Foragiven3.2.CompressiveSamplingMatchingPursuit 75precisionparameter,thealgorithmCoSaMPproducesans-sparseapproximation xthatsatises|x x|2 Cmax_,1s__x xs/2__1 +|e|2_wherexs/2isabest (s/2)-sparseapproximationtox. Therunningtimeis O(L log(|x|2/)), where Lboundsthecost of amatrixvectormultiplywithor.WorkingstorageisO(d).Remarks. 1. We note that as in the case of ROMP, CoSaMP requires knowledgeofthesparsitylevel s. AsdescribedinSection3.1.1, thereareseveral strategiestoestimatethislevel.2. Inthe hypotheses, aboundontherestrictedisometryconstant 2salsosuces. Indeed, Corollary3.2.7ofthesequel impliesthat4s 0.1holdswhenever2s 0.025.3. Theorem3.2.1isaresultof runningCoSaMPusinganiterativealgorithmtosolvetheleast-squaresstep. Weanalyzethisstepindetail below. Inthecaseofexactarithmetic, weagainanalyzeCoSaMPandprovideaniterationcountforthiscase:Theorem3.2.2 (Iteration Count).Suppose that CoSaMP is implemented with exactarithmetic. Afteratmost6(s + 1)iterations,CoSaMPproducesans-sparseapprox-imation xthatsatises|x a|2 20,whereistheunrecoverableenergy(3.2.1).SeeTheorem3.2.22inSection3.2.3belowformoredetails.Thealgorithmproducesans-sparseapproximationwhose2erroriscomparablewiththescaled1error of thebest (s/2)-sparseapproximationtothesignal. Of3.2.CompressiveSamplingMatchingPursuit 76course,the algorithm cannot resolvethe uncertaintydueto the additivenoise,sowealso pay for the energy in the noise. This type of error bound is structurally optimal,asdiscusswhendescribingtheunrecoverableenergybelow. Somedisparityinthesparsity levels (here, s versus s/2) seems to be necessary when the recovery algorithmiscomputationallyecient[58].Toproveourtheorem, weshowthatCoSaMPmakessignicantprogressduringeach iteration where the approximation error is large relative to unrecoverableenergyinthe signal. Thisquantitymeasuresthebaselineerror in our approximationthatoccursbecauseofnoiseinthesamplesorbecausethesignal isnotsparse. Forourpurposes,wedenetheunrecoverableenergybythefollowing.= |x xs|2 +1s |x xs|1 +|e|2. (3.2.1)The expression (3.2.1) for the unrecoverable energy can be simplied using Lemma 7from [32],whichstatesthat,for everysignaly CNand everypositiveintegert,wehave|y yt|2 12t|y|1.Choosingy = x xs/2andt = s/2,wereach 1.71s__x xs/2__1 +|e|2. (3.2.2)Inwords, theunrecoverableenergyiscontrolledbythescaled1normofthesignaltail.Thetermunrecoverableenergyisjustiedbyseveralfacts. First,wemustpayforthe2errorcontaminatingthesamples. Tocheckthispoint,deneS= suppxs.The matrix S is nearly an isometry fromS2to m2 , so an error in the large components3.2.CompressiveSamplingMatchingPursuit 77of the signal inducesan error of equivalent size in the samples. Clearly, we can neverresolvethisuncertainty.The term s1/2|x xs|1is also required on account of classical results about theGelfandwidthsof thed1ball ind2, duetoKashin[42] andGarnaevGluskin[30].In the language of compressivesampling, their work has the following interpretation.Let be a xed md sampling matrix. Suppose that, for every signal x Cd, thereisanalgorithmthatusesthesamplesu = xtoconstructanapproximationathatachievestheerrorbound|x a|2 Cs |x|1.Thenthenumbermofmeasurementsmustsatisfym cs log(d/s).3.2.3 ProofsofTheoremsTheorem3.2.1willbeshownbydemonstratingthat thefollowingiteration invariantholds. Theseresultscanbefoundin[53].Theorem3.2.3(IterationInvariant). Foreachiterationk 0,thesignal approxi-mationakiss-sparseand__x ak+1__2 0.5__x ak__2 + 10.Inparticular,__x ak__2 2k|x|2 + 20.Wewillrstshowthisholdsfor sparseinputsignals,andthenderivethegeneralcase.Whenthesamplingmatrix satisestherestrictedisometryinequalities(2.1.4),it3.2.CompressiveSamplingMatchingPursuit 78has several other properties that we require repeatedly in the proof that the CoSaMPalgorithm is correct. Our rst observation is a simple translation of(2.1.4) into otherterms,inthesamelightasProposition3.1.6usedintheproofofROMP.Proposition3.2.4. Supposehasrestrictedisometryconstantr. Let Tbeasetofrindicesorfewer. Then__Tu__2 _1 +r|u|2__Tu__2 11 r|u|2__TTx__2(1 r) |x|2__(TT)1x__2 11 r|x|2.wherethelast twostatementscontainanupperandlowerbound, dependingonthesignchosen.Proof. Therestrictedisometryinequalities(2.1.4)implythatthesingularvaluesofTlie between1 rand1 +r. The bounds follow from standard relationshipsbetween the singular values of Tand the singular values of basic functions of T.Asecondconsequenceisthatdisjointsetsofcolumnsfromthesamplingmatrixspannearlyorthogonalsubspaces. Thefollowingresultquantiesthisobservation.Proposition 3.2.5 (Approximate Orthogonality).Suppose has restricted isometryconstantr. LetSandTbedisjointsetsofindiceswhosecombinedcardinalitydoesnotexceedr. Then|ST| r.Proof. AbbreviateR = S T,andobservethatSTisasubmatrixofRRId.3.2.CompressiveSamplingMatchingPursuit 79Thespectral normofasubmatrixneverexceedsthenormoftheentirematrix. Wediscernthat|ST| |RR Id| max(1 +r) 1, 1 (1 r) = rbecausetheeigenvaluesofRRliebetween1 rand1 +r.Thisresultwillbeappliedthroughthefollowingcorollary.Corollary3.2.6. Supposehasrestrictedisometryconstantr. LetTbeasetofindices,andletxbeavector. Providedthatr [T suppx[,|Tx[Tc|2 r|x[Tc|2.Proof. DeneS= suppx T,sowehavex[S= x[Tc . Thus,|Tx[Tc|2= |Tx[S|2 |TS| |x[S|2 r|x[Tc|2,owingtoProposition3.2.5.Asasecondcorollary, weshowthat2rgivesweakcontrol over thehigher re-strictedisometryconstants.Corollary3.2.7. Letcandrbepositiveintegers. Thencr c2r.Proof. Theresult is clearlytruefor c =1, 2, sowe assume c 3. Let Sbeanarbitraryindexset of sizecr, andlet M=SS Id. It suces tocheckthat|M| c2r. Tothatend, webreakthematrixMintorrblocks, whichwedenoteMij. Ablockversionof Gershgorinstheoremstatesthat |M|satisesat3.2.CompressiveSamplingMatchingPursuit 80leastoneoftheinequalities[|M| |Mii|[

j=i|Mij| wherei = 1, 2, . . . , c.Thederivationisentirelyanalogouswiththeusual proof of Gershgorinstheorem,soweomit thedetails. Foreachdiagonal block, wehave |Mii| rbecauseofthe restrictedisometryinequalities (2.1.4). For eacho-diagonal block, we have|Mij| 2rbecauseof Proposition3.2.5. SubstitutetheseboundsintotheblockGershgorintheoremandrearrangetocompletetheproof.Finally, we present a result that measures how much the sampling matrix inatesnonsparsevectors. Thisboundpermitsustoestablishthemajorresultsforsparsesignalsandthentransfertheconclusionstothegeneralcase.Proposition3.2.8(EnergyBound). Supposethatveriestheupperinequalityof(2.1.4),viz.|x|2 _1 +r|x|2when |x|0 r.Then,foreverysignalx,|x|2 _1 +r_|x|2 +1r |x|1_.Proof. First, observe that thehypothesis of thepropositioncanberegardedas astatementabouttheoperatornormofasamapbetweentwoBanachspaces. ForasetI 1, 2, . . . , N,writeBI2fortheunitballin2(I). DenetheconvexbodyS= conv__|I|rBI2_CN,3.2.CompressiveSamplingMatchingPursuit 81andnoticethat,byhypothesis,theoperatornorm||S2 = maxxS|x|2 _1 +r.DeneasecondconvexbodyK=_x : |x|2 +1r |x|1 1_CN,andconsidertheoperatornorm||K2 = maxxK|x|2.Thecontentofthepropositionistheclaimthat||K2 ||S2.Toestablishthispoint,itsucestocheckthatK S.Chooseavectorx K. Wepartitionthesupportofxintosetsofsizer. LetI0indextherlargest-magnitudecomponentsofx, breakingtieslexicographically. LetI1indexthenextlargestrcomponents, andsoforth. Notethatthenal blockIJmayhavefewerthanrcomponents. Wemayassumethatx[Ijisnonzeroforeachj.Thispartitioninducesadecompositionx = x[I0 +

Jj=0x[Ij= 0y0 +

Jj=0jyjwherej=__x[Ij__2and yj= 1jx[Ij.3.2.CompressiveSamplingMatchingPursuit 82Byconstruction, eachvectoryjbelongstoSbecauseitisr-sparseandhasunit2norm. We will prove that

j j 1, which implies that x can be written as a convexcombinationof vectorsfromthesetS. Asaconsequence, x S. ItemergesthatK S.Fixj intherange 1, 2, . . . , J. ItfollowsthatIjcontainsatmostrelementsandIj1containsexactlyrelements. Therefore,j=__x[Ij__2 r__x[Ij__ r 1r__x[Ij1__1where the last inequality holds because the magnitude of x on the set Ij1dominatesitslargestentryinIj. Summingtheserelations,weobtain

Jj=1j 1r

Jj=1__x[Ij1__1=1r |x|1.Itisclearthat0= |x[I0|2 |x|2. Wemayconcludethat

Jj=0j |x|2 +1r |x|1 1becausex K.IterationInvariant: SparseCaseWenowcommencetheproofofTheorem3.2.3. Forthemoment,letusassumethatthesignalisactuallysparse. Wewillremovethisassumptionlater.Theresultstatesthateachiterationofthealgorithmreducestheapproximationerrorbyaconstant factor, whileaddingasmall multipleof thenoise. Asacon-sequence, whentheapproximationerrorislargeincomparisonwiththenoise, the3.2.CompressiveSamplingMatchingPursuit 83algorithmmakessubstantialprogressinidentifyingtheunknownsignal.Theorem3.2.9(IterationInvariant: SparseCase). Assumethatxiss-sparse. Foreachk 0,thesignalapproximationakiss-sparse,and__x ak+1__2 0.5__x ak__2 + 7.5 |e|2.Inparticular,__x ak__2 2k|x|2 + 15 |e|2.Theargumentproceedsinasequenceof shortlemmas, eachcorrespondingtoonestepinthealgorithm. Throughoutthissection, weretaintheassumptionthatxiss-sparse.Fixaniterationk 1. Wewritea=ak1forthesignal approximationatthebeginningoftheiteration. Denetheresidualr=x a, whichweinterpretasthepartofthesignalwehavenotyetrecovered. Sincetheapproximationaisalwayss-sparse, the residual r must be 2s-sparse. Notice that the vector v of updated samplescanbeviewedasnoisysamplesoftheresidual:vdef=u a = (x a) +e = r +e.Theidenticationphaseproducesasetofcomponentswheretheresidual signalstillhasalotofenergy.Lemma3.2.10(Identication). Theset=suppy2s, wherey=visthesignalproxy,containsatmost2sindices,and|r[c|2 0.2223 |r|2 + 2.34 |e|2.3.2.CompressiveSamplingMatchingPursuit 84Proof. Theidenticationphaseformsaproxyy=vfortheresidual signal. Thealgorithm thenselectsaset of2s componentsfrom ythat havethe largestmagni-tudes. Thegoaloftheproofistoshowthattheenergyintheresidualonthesetcissmallincomparisonwiththetotalenergyintheresidual.DenethesetR = suppr. SinceRcontainsatmost2s elements,ourchoiceofensuresthat |y[R|2 |y[|2. BysquaringthisinequalityandcancelingthetermsinR ,wediscoverthat__y[R\__2 __y[\R__2.Sincethecoordinatesubsets herecontainfewelements, wecanusetherestrictedisometryconstantstoprovideboundsonbothsides.First, observe that the set R contains at most 2s elements. Therefore, we mayapplyProposition3.2.4andCorollary3.2.6toobtain__y[\R__2=__\R(r +e)__2__\Rr__2 +__\Re__2 4s|r|2 +_1 +2s|e|2.Likewise,the set R contains 2s elements or fewer, so Proposition 3.2.4 and Corol-lary3.2.6yield__y[R\__2=__R\(r +e)__2__R\r[R\__2 __R\r[__2__R\e__2 (1 2s)__r[R\__2 2s|r|2_1 +2s|e|2.Sincetheresidual issupportedonR,wecanrewriter[R\=r[c. Finally, combine3.2.CompressiveSamplingMatchingPursuit 85thelastthreeinequalitiesandrearrangetoobtain|r[c|2 (2s +4s) |r|2 + 21 +2s|e|21 2s.Invokethenumericalhypothesisthat2s 4s 0.1tocompletetheargument.Thenextstepofthealgorithmmergesthesupportofthecurrentsignalapprox-imationawiththenewlyidentiedsetof components. Thefollowingresultshowsthatcomponentsofthesignalxoutsidethissethaveverylittleenergy.Lemma3.2.11(SupportMerger). Let beaset of at most 2sindices. ThesetT= suppacontainsatmost3sindices,and|x[Tc|2 |r[c|2.Proof. Sincesuppa T,wendthat|x[Tc|2= |(x a)[Tc|2= |r[Tc|2 |r[c|2,wheretheinequalityfollowsfromthecontainmentTc c.The estimationstepof thealgorithmsolves aleast-squares problemtoobtainvalues for the coecientsin the setT. We needa bound on the error of this approx-imation.Lemma3.2.12(Estimation). Let Tbeaset of at most3sindices, anddenetheleast-squaressignalestimatebbytheformulaeb[T= Tu and b[Tc= 0,3.2.CompressiveSamplingMatchingPursuit 86whereu = x +e. Then|x b|2 1.112 |x[Tc|2 + 1.06 |e|2.Thisresultassumesthatwesolvetheleast-squaresproblemininniteprecision.Inpractice, theright-handsideof theboundcontainsanextratermowingtotheerror from the iterative least-squaressolver. Below, we study how many iterations oftheleast-squaressolverarerequiredtomaketheleast-squareserrornegligibleinthepresentargument.Proof. Noterstthat|x b|2 |x[Tc|2 +|x[T b[T|2.Usingtheexpressionu = x +eandthefactTT= IdT,wecalculatethat|x[T b[T|2=__x[T T(x[T+ x[Tc +e)__2=__T(x[Tc +e)__2__(TT)1Tx[Tc__2 +__Te__2.The cardinalityof Tis at most 3s, and x is s-sparse,soProposition 3.2.4 and Corol-lary3.2.6implythat|x[T b[T|2 11 3s|Tx[Tc|2 +11 3s|e|24s1 3s|x[Tc|2 +|e|21 3s.3.2.CompressiveSamplingMatchingPursuit 87Combinetheboundstoreach|x b|2 _1 +4s1 3s_|x[Tc|2 +|e|21 3s.Finally,invokethehypothesisthat3s 4s 0.1.Thenalstepofeachiterationistoprunetheintermediateapproximationtoitslargeststerms. Thefollowinglemmaprovidesaboundontheerrorintheprunedapproximation.Lemma3.2.13(Pruning). Theprunedapproximationbssatises|x bs|2 2 |x b|2.Proof. Theintuitionisthatbsisclosetob,whichisclosetox. Rigorously,|x bs|2 |x b|2 +|b bs|2 2 |x b|2.Thesecondinequalityholdsbecausebsisthebests-sparseapproximationtob. Inparticular,thes-sparsevectorxisaworseapproximation.Wenowcompletetheproof of theiterationinvariant forsparsesignals, Theo-rem3.2.9. At theendof aniteration, thealgorithmforms anewapproximationak=bs, whichisevidentlys-sparse. Applyingthelemmaswehaveestablished, we3.2.CompressiveSamplingMatchingPursuit 88easilyboundtheerror:__x ak__2= |x bs|2 2 |x b|2Pruning(Lemma3.2.13) 2(1.112 |x[Tc|2 + 1.06 |e|2) Estimation(Lemma3.2.12) 2.224 |r[c|2 + 2.12 |e|2Supportmerger(Lemma3.2.11) 2.224(0.2223 |r|2 + 2.34 |e|2) + 2.12 |e|2Identication(Lemma3.2.10)< 0.5 |r|2 + 7.5 |e|2= 0.5__x ak1__2 + 7.5 |e|2.ToobtainthesecondboundinTheorem3.2.9, simplysolvetheerrorrecursionandnotethat(1 + 0.5 + 0.25 +. . . )7.5 |e|2 15 |e|2.Thispointcompletestheargument.Beforeextendingtheiterationinvariant tothesparsecase, werst analyzeindetailtheleast-sqauresstep. Thiswillallowustocompletelyproveour mainresult,Theorem3.2.1.LeastSquaresAnalysisTodevelopanecientimplementationofCoSaMP, itiscritical touseaniterativemethodwhenwesolvetheleast-squaresproblemintheestimationstep. Hereweanalyzethisstepforthenoise-freecase. Thetwonatural choicesareRichardsonsiteration and conjugate gradient. The ecacy of these methods rests on the assump-tion that the sampling operator has small restricted isometry constants. Indeed, since3.2.CompressiveSamplingMatchingPursuit 89thesetTconstructedinthesupportmergerstepcontainsatmost3scomponents,thehypothesis4s 0.1ensuresthattheconditionnumber(TT) =max(TT)min(TT) 1 +3s1 3s< 1.223.This conditionnumber is closelyconnectedwiththeperformanceof Richardsonsiterationandconjugategradient. Inthissection,weshowthatTheorem3.2.9holdsifweperformaconstantnumberofiterationsofeitherleast-squaresalgorithm.For completeness, let us explainhowRichardsons iterationcanbeappliedtosolvethe least-squaresproblemsthat arise in CoSaMP. Supposewe wishtocomputeAu where A is a tall, full-rank matrix. Recalling the denition of the pseudoinverse,werealizethatthisamountstosolvingalinearsystemoftheform(AA)b = Au.ThisproblemcanbeapproachedbysplittingtheGrammatrix:AA = Id +MwhereM = AAId. Givenan initialiteratez0,Richardonsmethodproducesthesubsequentiteratesviatheformulaz+1= Au Mz.Evidently, thisiterationrequiresonlymatrixvectormultiplieswithAandA. ItisworthnotingthatRichardsonsmethodcanbeaccelerated[2, Sec. 7.2.5], butweomitthedetails.3.2.CompressiveSamplingMatchingPursuit 90ItisquiteeasytoanalyzeRichardsonsiteration[2,Sec.7.2.1]. Observethat__z+1Au__2=__M(zAu)__2 |M|__zAu__2.Thisrecursiondelivers__zAu__2 |M|__z0Au__2for = 0, 1, 2, . . . .Inwords,theiterationconvergeslinearly.Inoursetting, A=TwhereTisasetof atmost3sindices. Therefore, therestrictedisometryinequalities(2.1.4)implythat|M| = |TT Id| 3s.Wehaveassumedthat3s 4s 0.1, whichmeansthattheiterationconvergesquite fast. Once again, the restricted isometry behavior of the sampling matrix playsanessentialroleintheperformanceoftheCoSaMPalgorithm.Conjugategradientprovidesevenbetterguaranteesforsolvingtheleast-squaresproblem,butitissomewhatmorecomplicatedtodescribeandrathermorediculttoanalyze. Wereferthereaderto[2, Sec. 7.4] formoreinformation. Thefollow-inglemmasummarizes thebehavior of bothRichardsons iterationandconjugategradientinoursetting.Lemma3.2.14(ErrorBoundforLS). Richardsonsiterationproducesasequencezofiteratesthatsatisfy__zTu__2 0.1

__z0Tu__2for = 0, 1, 2, . . . .3.2.CompressiveSamplingMatchingPursuit 91Conjugategradientproducesasequenceofiteratesthatsatisfy__zTu__2 2

__z0Tu__2for = 0, 1, 2, . . . .where =_(TT) 1_(TT) + 1 0.072.ThiscanevenbeimprovedfurtheriftheeigenvaluesofTTareclustered[64].Iterativeleast-squaresalgorithmsmustbeseededwithaninitial iterate, andtheirperformancedependsheavilyonawiseselectionthereof. CoSaMPoersanaturalchoicefortheinitializer: thecurrentsignal approximation. Asthealgorithmpro-gresses, the current signal approximation provides an increasingly good starting pointfor solving the least-squares problem. The following shows that the error in the initialiterateiscontrolledbythecurrentapproximationerror.Lemma 3.2.15 (Initial Iterate for LS).Let x be an s-sparse signal with noisy samplesu = x+e. Let ak1be thesignal approximation at the end of the (k 1)th iteration,andletTbethesetofcomponentsidentiedbythesupportmerger. Then__ak1Tu__2 2.112__x ak1__2 + 1.06 |e|2Proof. ByconstructionofT,theapproximationak1issupportedinsideT,so__x[Tc__2=__(x ak1)[Tc__2 __x ak1__2.UsingLemma3.2.12, wemaycalculatehowfarak1liesfromthesolutiontothe3.2.CompressiveSamplingMatchingPursuit 92least-squaresproblem.__ak1Tu__2 __x ak1__2 +__x Tu__2__x ak1__2 + 1.112__x[Tc__2 + 1.06 |e|2 2.112__x ak1__2 + 1.06 |e|2.We needtodetermine howmanyiterations of the least-squares algorithmarerequiredtoensurethattheapproximationproducedissucientlygoodtosupporttheperformanceofCoSaMP.Corollary3.2.16(EstimationbyIterativeLS). Supposethat weinitializetheLSalgorithmwithz0= ak1. Afteratmostthreeiterations,bothRichardsonsiterationandconjugategradientproduceasignalestimatebthatsatises__x b__2 1.112__x[Tc__2 + 0.0022__x ak1__2 + 1.062 |e|2.Proof. Combine Lemma 3.2.14andLemma 3.2.15tosee that three iterations ofRichardsonsmethodyield__z3Tu__2 0.002112__x ak1__2 + 0.00106 |e|2.Theboundforconjugategradientisslightlybetter. Letb[T= z3. Accordingtotheestimationresult,Lemma3.2.12,wehave__x Tu__2 1.112__x[Tc__2 + 1.06 |e|2.3.2.CompressiveSamplingMatchingPursuit 93Anapplicationofthetriangleinequalitycompletestheargument.Finally, weneedtocheckthatthesparseiterationinvariant, Theorem3.2.9stillholdswhenweuseaniterativeleast-squaresalgorithm.Theorem 3.2.17 (Sparse Iteration Invariant with Iterative LS).Suppose that we useRichardsonsiterationorconjugategradientfortheestimationstep, initializingtheLS algorithm with the current approximation ak1and performing three LS iterations.ThenTheorem3.2.9stillholds.Proof. WerepeatthecalculationfromtheabovecaseusingCorollary3.2.16insteadofthesimpleestimationlemma. Tothatend, recallthattheresidual r= x ak1.Then__x ak__2 2 |x b|2 2(1.112 |x[Tc|2 + 0.0022 |r|2 + 1.062 |e|2) 2.224 |r[c|2 + 0.0044 |r|2 + 2.124 |e|2 2.224(0.2223 |r|2 + 2.34 |e|2) + 0.0044 |r|2 + 2.124 |e|2< 0.5 |r|2 + 7.5 |e|2= 0.5__x ak1__2 + 7.5 |e|2.Thisboundispreciselywhatisrequiredforthetheoremtohold.Wearenowpreparedtoextendtheiterationinvarianttothegeneral case, andproveTheorem3.2.1.3.2.CompressiveSamplingMatchingPursuit 94ExtensiontoGeneral SignalsInthissection, wenallycompletetheproofofthemainresultforCoSaMP,Theo-rem 3.2.3. The remaining challenge is to remove the hypothesis that the target signalissparse, whichweframedinTheorems3.2.9and3.2.17. Althoughthisdicultymightseemlarge, thesolutionissimpleandelegant. Itturnsoutthatwecanviewthe noisy samples of a general signal as samples of a sparse signal contaminated withadierentnoisevectorthatimplicitlyreectsthetailoftheoriginalsignal.Lemma3.2.18(ReductiontoSparseCase). Let xbeanarbitraryvectorinCN.Thesamplevectoru = x +ecanalsobeexpressedasu = xs +ewhere| e|2 1.05_|x xs|2 +1s|x xs|1_+|e|2.Proof. Decomposex = xs +(x xs) toobtain u = xs + e where e = (x xs) +e.Tocomputethesizeoftheerrorterm, wesimplyapplythetriangleinequalityandProposition3.2.8:|e|2 _1 +s_|x xs|2 +1s|x xs|1_+|e|2.Finally,invokethefactthats 4s 0.1toobtain1 +s 1.05.ThislemmaisjustthetoolwerequiretocompletetheproofofTheorem3.2.3.ProofofTheorem3.2.3. Letxbeageneral signal, anduseLemma3.2.18towritethe noisyvector of samples u =xs+ e. Applythe sparse iterationinvariant,Theorem3.2.9,ortheanalogforiterativeleast-squares,Theorem3.2.17. Weobtain__xsak+1__2 0.5__xsak__2 + 7.5 | e|2.3.2.CompressiveSamplingMatchingPursuit 95Invoketheloweranduppertriangleinequalitiestoobtain__x ak+1__2 0.5__x ak__2 + 7.5 |e|2 + 1.5 |x xs|2.Finally,recalltheestimatefor | e|2fromLemma3.2.18,andsimplifytoreach__x ak+1__2 0.5__x ak__2 + 9.375 |x xs|2 + 7.875s|x xs|1 + 7.5 |e|2< 0.5__x ak__2 + 10.whereistheunrecoverableenergy(3.2.1).Wehavenowcollectedallthematerialweneedtoestablishthemainresult. Fixa precision parameter . After at most O(log(|x|2/)) iterations, CoSaMP producesans-sparseapproximationathatsatises|x a|2 C( +)in consequence of Theorem 3.2.3. Apply inequality (3.2.2) to bound the unrecoverableenergyintermsofthe1norm. Weseethattheapproximationerrorsatises|x a|2 Cmax_,1s__x xs/2__1 +|e|2_.According to Theorem 3.2.23, each iteration of CoSaMP is completed in time O(L),where Lboundsthecostofamatrixvectormultiplicationwithor . Thetotalruntime,therefore,isO(Llog(|x|2/)). ThetotalstorageisO(N).Finally, inthestatement of thetheorem, wereplace4swith2sbymeans ofCorollary3.2.7,whichstatesthatcr c2rforanypositiveintegerscandr.3.2.CompressiveSamplingMatchingPursuit 96IterationCountforExactArithmeticAspromised, wenowprovideaniterationcountforCoSaMPinthecaseof exactarithmetic. Theseresultscanbefoundin[52].Weobtainanestimateonthenumber of iterations of theCoSaMPalgorithmnecessarytoidentifythe recoverableenergyin asparsesignal,assumingexactarith-metic. Except where stated explicitly, we assume that x is s-sparse. It turns out thatthenumberofiterationsdependsheavilyonthesignalstructure. Letusexplaintheintuitionbehindthisfact.Whentheentriesin thesignaldecayrapidly,the algorithm mustidentifyandre-move the largest remaining entry from the residual before it can make further progresson the smaller entries. Indeed, the large component in the residual contaminates eachcomponentofthesignalproxy. Inthiscase, thealgorithmmayrequireaniterationormoretondeachcomponentinthesignal.On the other hand, when the s entries of the signal are comparable, the algorithmcansimultaneouslylocatemanyentries just byreducingthenormof theresidualbelowthemagnitudeofthesmallestentry. Sincethelargestentryofthesignalhasmagnitude at least s1/2times the2norm of the signal,the algorithm can nd all scomponentsofthesignalafteraboutlog siterations.To quantifythese intuitions,we want to collect the components of the signal intogroups that are comparable with each other. To that end, dene the component bandsofasignalxbytheformulaeBjdef=_i : 2(j+1)|x|22< [xi[2 2j|x|22_forj= 0, 1, 2, . . . . (3.2.3)3.2.CompressiveSamplingMatchingPursuit 97Theproleofthesignalisthenumberofbandsthatarenonempty:prole(x)def= #j: Bj ,= .Inwords, theprolecountshowmanyordersofmagnitudeatwhichthesignal hascoecients. Itisclearthattheproleofans-sparsesignalisatmosts.SeeFigure3.2.1forimagesofstylizedsignalswithdierentproles.0 5 10 15 20 25 3000.10.20.30.40.50.60.70.80.910 5 10 15 20 25 3000.10.20.30.40.50.60.70.80.91Figure 3.2.1: Illustration of two unit-norm signals with sharply dierent proles (left:low,right: high).First, we prove a result on the number of iterations needed to acquire an s-sparsesignal. At the end of the section, we extend this result to general signals, which yieldsTheorem3.2.2.Theorem3.2.19(IterationCount: SparseCase). Letxbeans-sparsesignal,anddenep = prole(x). Afteratmostp log4/3(1 + 4.6_s/p) + 63.2.CompressiveSamplingMatchingPursuit 98iterations,CoSaMPproducesanapproximationathatsatises|x a|2 17 |e|2.For axeds,theboundon thenumberofiterationsachievesitsmaximumvalueatp = s. Sincelog4/3 5.6 < 6,thenumberofiterationsneverexceeds6(s + 1).Letusinstatesomenotationthatwill bevaluableintheproof of thetheorem.Wewritep = prole(x). Foreachk= 0, 1, 2, . . . ,thesignalakistheapproximationafter the kth iteration. We abbreviate Sk= suppak, and we dene the residual signalrk= x ak. Thenormoftheresidualcanbeviewedastheapproximationerror.Foranonnegativeintegerj,wemaydeneanauxiliarysignalyjdef=x[Sij BiInotherwords, yjisthepartofxcontainedinthebandsBj, Bj+1, Bj+2, . . . . Foreachj J,wehavetheestimate__yj__22

ij2i|x|22[Bi[ (3.2.4)bydenitionofthebands. Theseauxiliarysignalsplayakeyroleintheanalysis.The proof of the theoreminvolves asequence of lemmas. The rst object istoestablishanalternativethatholdsineachiteration. Onepossibilityisthattheapproximationerrorissmall,whichmeansthatthealgorithmiseectivelynished.Otherwise, theapproximationerrorisdominatedbytheenergyintheunidentiedpart of the signal, and the subsequent approximation error is a constant factor smaller.Lemma 3.2.20. For eachiterationk =0, 1, 2, . . . , at least one of the following3.2.CompressiveSamplingMatchingPursuit 99alternativesholds. Either__rk__2 70 |e|2(3.2.5)orelse__rk__2 2.3__x[Sck__2and (3.2.6)__rk+1__2 0.75__rk__2. (3.2.7)Proof. DeneTkasthe mergedsupport that occursduringiterationk. ThepruningstepensuresthatthesupportSkoftheapproximationattheendoftheiterationisasubsetofthemergedsupport,so__x[Tck__2 __x[Sck__2fork = 1, 2, 3, . . . .At the end of the kth iteration, the pruned vector bsbecomes the next approximationak, sotheestimationandpruningresults,Lemmas3.2.12and3.2.13,together implythat__rk__2 2(1.112__x[Tck__2 + 1.06 |e|2) 2.224__x[Sck__2 + 2.12 |e|2fork = 1, 2, 3, . . . . (3.2.8)Notethatthesamerelationholdstriviallyforiterationk=0becauser0=xandS0= .Supposethatthereisaniterationk 0where__x[Sck__2< 30 |e|2.3.2.CompressiveSamplingMatchingPursuit 100Wecanintroducethisbounddirectlyintotheinequality(3.2.8)toobtaintherstconclusion(3.2.5).Supposeonthecontrarythatiniterationkwehave__x[Sck__2 30 |e|2.Introducingthisrelationintotheinequality(3.2.8)leadsquicklytotheconclusion(3.2.6). Wealsohavethechainofrelations__rk__2 __rk[Sck__2=__(x ak)[Sck__2=__x[Sck__2 30 |e|2.Therefore, thesparseiterationinvariant, Theorem3.2.9ensuresthat(3.2.7)holds.Thenextlemmacontainsthecritical partof theargument. Under thesecondalternativeinthepreviouslemma,weshowthatthealgorithmcompletelyidentiesthesupportofthesignal,andweboundthenumberofiterationsrequiredtodoso.Lemma3.2.21. FixK= p log4/3(1 + 4.6_s/p). Assumethat (3.2.6)and(3.2.7)areinforceforeachiterationk= 0, 1, 2, . . . , K. ThensuppaK= suppx.Proof. First, we check that, once the norm of the residual is smaller than each elementof aband, thecomponentsinthatbandpersistinthesupportof eachsubsequentapproximation. DeneJtobethesetof nonemptybands, andxabandj J.Supposethat,forsomeiterationk,thenormoftheresidualsatises__rk__2 2(j+1)/2|x|2. (3.2.9)ThenitmustbethecasethatBj suppak. If not, thensomecomponenti Bj3.2.CompressiveSamplingMatchingPursuit 101appearsintheresidual: rki= xi. Thissuppositionimpliesthat__rk__2 [xi[ > 2(j+1)/2|x|2,anevident contradiction. Since (3.2.7) guarantees that the normof the residualdeclinesineachiteration, (3.2.9)ensures thatthesupportof eachsubsequent ap-proximationcontainsBj.Next, we boundthe number of iterations requiredtondthe next nonemptybandBj,giventhatwehavealreadyidentiedthebandsBiwherei < j. Formally,assumethat thesupport Skofthecurrentapproximationcontains Bifor eachi < j.Inparticular, thesetof missingcomponentsSck suppyj. Itfollowsfromrelation(3.2.6)that__rk__2 2.3__yj__2.WecanconcludethatwehaveidentiedthebandBjiniterationk +if__rk+__2 2(j+1)/2|x|2.Accordingto(3.2.7), we reduce the error byafactor of 1=0.75duringeachiteration. Therefore,thenumberofiterationsrequiredtoidentifyBjisatmostlog_2.3 |yj|22(j+1)/2|x|2_We discover that the total number of iterations required to identify all the (nonempty)bandsisatmostkdef=

jJlog_2.3 2(j+1)/2|yj|2|x|2_.3.2.CompressiveSamplingMatchingPursuit 102For eachiterationk k,itfollowsthatsuppak= suppx.It remains to bound kin terms of the prole p of the signal. For convenience,wefocus on a slightly dierent quantity. First, observe that p = [J[. Using the geometricmeanarithmeticmeaninequality,wediscoverthatexp_1p

jJlog_2.3 2(j+1)/2|yj|2|x|2__ exp_1p

jJlog_1 + 2.3 2(j+1)/2|yj|2|x|2__1p

jJ_1 + 2.3 2(j+1)/2|yj|2|x|2_= 1 +2.3p

jJ_2j+1|yj|22|x|22_1/2.To bound the remaining sum, we recall the relation (3.2.4). Then we invoke Jensensinequalityandsimplifytheresult.1p

jJ_2j+1|yj|22|x|22_1/21p

jJ_2j+1

ij2i[Bi[_1/2_1p

jJ2j+1

ij2i[Bi[_1/2_1p

i0[Bi[

ji2ji+1_1/2_4p

i0[Bi[_1/2= 2_s/p.The nal equality holds because the total number of elements in all the bands equalsthesignalsparsitys. Combiningthesebounds,wereachexp_1p

jJlog_2.3 2(j+1)/2|yj|2|x|2__ 1 + 4.6_s/p.Takelogarithms, multiplybyp,anddividethroughbylog . Weconcludethatthe3.2.CompressiveSamplingMatchingPursuit 103requirednumberofiterationskisboundedask p log(1 + 4.6_s/p).Thisistheadvertisedconclusion.Finally, we check that the algorithm produces a small approximation error withinareasonablenumberofiterations.ProofofTheorem3.2.19. Abbreviate K= p log(1+4.6_s/p). Suppose that (3.2.5)neverholdsduringtherstKiterationsofthealgorithm. Underthiscircumstance,Lemma3.2.20 impliesthat both (3.2.6) and (3.2.7) hold duringeach of theseKiter-ations. It follows from Lemma 3.2.21 that the support SKof the Kth approximationequals the support of x. Since SKis contained in the merged support TK, we see thatthevectorx[TcK=0. Therefore, theestimationandpruningresults, Lemmas3.2.12and3.2.13,showthat__rK__2 2 _1.112__x[TcK__2 + 1.06 |e|2_= 2.12 |e|2.Thisestimatecontradicts(3.2.5).Itfollowsthatthereisaniterationk Kwhere(3.2.5)isinforce. Repeatedapplicationsoftheiterationinvariant,Theorem3.2.9,allowustoconcludethat__rK+6__2< 17 |e|2.Thispointcompletestheargument.Finally,weextendthesparseiterationcountresulttothegeneralcase.3.2.CompressiveSamplingMatchingPursuit 104Theorem3.2.22(IterationCount). Let xbeanarbitrarysignal, anddenep=prole(xs). Afteratmostp log4/3(1 + 4.6_s/p) + 6iterations,CoSaMPproducesanapproximationathatsatises|x a|2 20 |e|2.Proof. Letxbeageneral signal, andletp=prole(xs). Lemma3.2.18allowsustowritethenoisyvectorofsamplesu = xs + e. Thesparseiterationcountresult,Theorem3.2.19,statesthatafteratmostp log4/3(1 + 4.6_s/p) + 6iterations,thealgorithmproducesanapproximationathatsatises|xs a|2 17 |e|2.Applythelowertriangleinequalitytotheleft-handside. Thenrecall theestimateforthenoiseinLemma3.2.18,andsimplifytoreach|x a|2 17 | e|2 +|x xs|2 18.9 |x xs|2 +17.9s |x xs|1 + 17 |e|2< 20,whereistheunrecoverableenergy.3.2.CompressiveSamplingMatchingPursuit 105ProofofTheorem3.2.2. Invoke Theorem3.2.22. Recall that the estimate for thenumber of iterations is maximized with p = s, which gives an upper bound of 6(s+1)iterations,independentofthesignal.3.2.4 ImplementationandRuntimeCoSaMPwas designedtobeapractical methodfor signal recovery. Anecientimplementationofthealgorithmrequiressomeideasfromnumerical linearalgebra,as well as some basic techniques from the theory of algorithms. This section discussesthe key issues and develops an analysis of the running time for the two most commonscenarios.Wefocusontheleast-squaresproblemintheestimationstepbecauseitisthemajor obstacletoafast implementationofthe algorithm. Thealgorithm guaranteesthatthematrixTneverhasmorethan3scolumns, soourassumption4s 0.1impliesthatthematrixTisextremelywellconditioned. Asaresult,wecanapplythepseudoinverseT=(TT)1Tveryquicklyusinganiterativemethod, suchasRichardsons iteration[2, Sec. 7.2.3] or conjugategradient [2, Sec. 7.4]. Thesetechniqueshave the additional advantage that they only interact with the matrix Tthrough its action on vectors. It follows that the algorithm performs better when thesamplingmatrixhasafastmatrixvectormultiply.Let us stress that direct methods for least squaresare likelyto be extremelyinef-cient in this setting. The rst reason is that each least-squares problem may containsubstantiallydierentsetsofcolumnsfrom. Asaresult, itbecomesnecessarytoperformacompletelynewQRorSVDfactorizationduringeachiterationatacostofO(s2m). Thesecondproblemisthatcomputingthesefactorizationstypicallyre-quires direct access tothecolumns of thematrix, whichis problematicwhenthe3.2.CompressiveSamplingMatchingPursuit 106matrix isaccessedthrough itsaction on vectors. Third,directmethodshavestoragecostsO(sm),whichmaybedeadlyforlarge-scaleproblems.The remaining steps of the algorithm involve standard techniques. Let us estimatetheoperationcounts.ProxyFormingtheproxyisdominatedbythecostof thematrixvectormultiplyv.IdenticationWecanlocatethelargest2sentriesofavectorintimeO(N)usingtheapproachin[11,Ch.9]. Inpractice, itmaybefastertosorttheentriesofthe signal in decreasingorder of magnitudeat costO(N log N)and then selectthe rst2s ofthem. Thelatterprocedurecan beaccomplishedwithquicksort,mergesort, orheapsort[11,Sec. II]. Toimplementthealgorithmtotheletter,the sorting method needs to be stable because we stipulate that ties are brokenlexicographically. Thispointisnotimportantinpractice.SupportMerger WecanmergetwosetsofsizeO(s)inexpectedtimeO(s)usingrandomized hashing methods [11, Ch. 11]. One can also sort both sets rst andusetheelementarymergeprocedure[11,p.29]foratotalcostO(s log s).LSestimationWe use Richardsons iterationor conjugate gradient to computeTu. Initializingthe least-squaresalgorithm requiresamatrixvectormultiplywith T. Each iteration of the least-squares method requires one matrixvectormultiply each with Tand T. Since Tis a submatrix of , the matrixvectormultipliescanalsobeobtainedfrommultiplicationwiththefull matrix. Weprovedabove that aconstant number of least-squares iterations suces forTheorem3.2.3tohold.PruningThisstepis similartoidentication. Pruningcan be implementedin time3.2.CompressiveSamplingMatchingPursuit 107O(s),butitmaybepreferabletosortthecomponentsofthevectorbymagni-tudeandthenselecttherstsatacostofO(s log s).SampleUpdateThis step is dominated by the cost of the multiplication of withthes-sparsevectorak.Table3.1summarizesthisdiscussionintwoparticularcases. Therstcolumnshows what happens when the sampling matrix is applied to vectors in the standardway, but wehaverandomaccess tosubmatrices. Thesecondcolumnshowswhathappenswhenthesamplingmatrixanditsadjointbothhaveafastmultiplywith cost L, where we assumethat L N. A typical value is L= O(N log N). Inparticular,apartialFouriermatrixsatisesthisbound.Table 3.1: Operation count for CoSaMP. Big-O notation is omitted for legibility. Thedimensionsofthe samplingmatrix are mN;the sparsitylevelis s. The numberLboundsthecostofamatrixvectormultiplywithor.Step Standardmultiply FastmultiplyFormproxy mN LIdentication N NSupportmerger s sLSestimation sm LPruning s sSampleupdate sm LTotalperiteration O(mN) O(L)Finally, we note that the storage requirements of the algorithm are also favorable.Asidefromthestoragerequiredbythesamplingmatrix, thealgorithmconstructsonlyonevector of lengthN, thesignal proxy. Thesamplevectors uandvhavelengthm, sotheyrequireO(m)storage. Thesignal approximationscanbestored3.2.CompressiveSamplingMatchingPursuit 108usingsparsedatastructures, sotheyrequireatmostO(s log N)storage. Similarly,theindexsetsthatappearrequireonlyO(s log N)storage. Thetotal storageisatworstO(N).Thefollowingresultsummarizesthisdiscussion.Theorem 3.2.23 (Resource Requirements).Each iteration of CoSaMP requires O(L)time,whereLboundsthecostofamultiplicationwiththematrixor. Theal-gorithmusesstorageO(N).AlgorithmicVariationsThissectiondescribesotherpossiblehaltingcriteriaandtheirconsequences. Italsoproposessomeothervariationsonthealgorithm.Thereare threenatural approachestohaltingthealgorithm. Therst,whichwehave discussedin the body of the paper, is to stop after a xed number of iterations.Another possibilityis to use the norm |v|2of the currentsamplesas evidenceaboutthenorm |r|2oftheresidual. Athirdpossibilityistousethemagnitude |y|oftheentriesoftheproxytoboundthemagnitude |r|oftheentriesoftheresidual.It suces to discuss halting criteria for sparse signals because Lemma 3.2.18 showsthatthegeneral casecanbeviewedintermsofsamplingasparsesignal. Letxbeans-sparsesignal, andletabeans-sparseapproximation. Theresidualr= x a.Wewritev=r + efortheinducednoisysamplesoftheresidual andy= vforthesignalproxy.Thediscussionproceedsintwosteps. First, wearguethatanapriori haltingcriterion will result in a guarantee about the quality of the nal signal approximation.3.2.CompressiveSamplingMatchingPursuit 109Theorem3.2.24(HaltingI). Thehaltingcriterion |v|2 ensuresthat|x a|2 1.06( +|e|2).Thehaltingcriterion |y| /2sensuresthat|x a| 1.12 + 1.17 |e|2.Proof. Sinceris2s-sparse,Proposition3.2.4ensuresthat_1 2s|r|2 |e|2 |v|2.If |v|2 ,itisimmediatethat|r|2 +|e|21 2s.Thedenitionr= x aandthenumerical bound2s 4s 0.1dispatchtherstclaim.LetR = suppr,andnotethat [R[ 2s. Proposition3.2.4resultsin(1 2s) |r|2 _1 +2s|e|2 |y[R|2.Since|y[R|2 2s |y[R| 2s |y|,wendthattherequirement |y| /2sensuresthat|r| +1 +2s|e|21 2s.3.2.CompressiveSamplingMatchingPursuit 110Thenumericalbound2s 0.1completestheproof.Second, wecheckthateachhaltingcriterionistriggeredwhentheresidual hasthedesiredproperty.Theorem3.2.25(Halting II).The haltingcriterion |v|2 is triggered as soon as|x a|2 0.95( |e|2).Thehaltingcriterion |y| /2sistriggeredassoonas|x a| 0.45s0.68 |e|2s.Proof. Proposition3.2.4showsthat|v|2 _1 +2s|r|2 +|e|2.Therefore,thecondition|r|2 |e|21 +2sensures that |v|2 . Note that 2s 0.1 to complete the rst part of the argument.Now let R be the singleton containing the index of a largest-magnitude coecientofy. Proposition3.2.4impliesthat|y|= |y[R|2 _1 +1|v|2.Bytherstpartofthistheorem,thehaltingcriterion |y| /2sistriggeredas3.2.CompressiveSamplingMatchingPursuit 111soonas|x a|2 0.95 _2s1 +1|e|2_.Sincex ais2s-sparse,wehavethebound |x a|2 2s |x a|. Towrapup,recallthat1 2s 0.1.Nextwediscussothervariationsofthealgorithm.Hereisaversionofthealgorithmthatis, perhaps, simplerthanthatdescribedabove. At eachiteration,weapproximatethecurrentresidualratherthan the entiresignal. ThisapproachissimilartoHHSpursuit[32]. Theinnerloopchangesinthefollowingmanner.IdenticationAsbefore,select = suppy2s.EstimationSolvealeast-squaresproblemwiththecurrentsamplesinsteadoftheoriginal samplestoobtainanapproximationof theresidual signal. Formally,b = v. Inthiscase,oneinitializestheiterativeleast-squaresalgorithmwiththe zero vector to take advantage of the fact that the residual is becoming small.ApproximationMerger Addthisapproximationof theresidual tothepreviousapproximationofthesignal toobtainanewapproximationofthesignal: c=ak1+b.PruningConstructthes-sparsesignalapproximation: ak= cs.SampleUpdateUpdatethesamplesasbefore: v = u ak.Byadaptingtheargument inthispaper, wehavebeenabletoshowthatthisalgorithm alsosatisesTheorem3.2.1. We believethisversionisquitepromisingforapplications.3.2.CompressiveSamplingMatchingPursuit 112Analternativevariationisasfollows. Aftertheinner loopof thealgorithmiscomplete, we cansolve another least-squares probleminaneort toimprove thenal result. If a is the approximation at the end of the loop, we set T= suppa. Thensolve b = Tu and output the s-sparse signal approximation b. Note that the outputapproximationisnotguaranteedtobebetterthanabecauseofthenoisevectore,butitshouldneverbemuchworse.Another variationis toprune the mergedsupport T downtosentries beforesolving the least-squares problem. One may use the values of the proxy y as surrogatesfor the unknown values of the new approximation on the set . Since the least-squaresproblemissolvedattheendoftheiteration,thecolumnsofthatareusedintheleast-squaresapproximationareorthogonal tothecurrentsamples v. Asaresult,the identicationstep always selectsnew componentsin each iteration. We havenotattemptedananalysisofthisalgorithm.3.2.5 NumericalResultsNoiselessNumericalStudiesThissectiondescribesourexperimentsthatillustratethesignal recoverypowerofCoSaMP. SeeSectionA.4for theMatlabcodeusedinthesestudies. Weexperi-mentallyexaminehowmany measurementsm are necessarytorecovervarious kindsofs-sparsesignalsinRdusingROMP. Wealsodemonstratethatthenumberofit-erationsCoSaMPneedstorecoverasparsesignal isinpracticeatmostlinearthesparsity,andinfactthisservesasasuccessfulhaltingcriterion.Firstwedescribethesetupofourexperiments. Formanyvaluesoftheambientdimensiond, thenumber of measurements m, andthe sparsitys, we reconstructrandom signals using CoSaMP. For each set of values, we generate an md Gaussian3.2.CompressiveSamplingMatchingPursuit 113measurement matrixandthenperform500independent trials. The results weobtainedusingBernoulli measurementmatriceswereverysimilar. Inagiventrial,we generate an s-sparse signal x in one of two ways. In either case, we rst select thesupportofthesignal bychoosingscomponentsuniformlyatrandom(independentfrom the measurement matrix ). In the cases where we wish to generate at signals,wethensetthesecomponentstoone. Inthecaseswherewewishtogeneratesparsecompressiblesignals,wesettheithcomponentofthesupporttoplusorminusi1/pforaspeciedvalueof0 < p < 1. WethenexecuteCoSaMPwiththemeasurementvectoru = x.Figure 3.2.2 depicts the percentage (from the 500 trials) of sparse at signals thatwerereconstructedexactly. Thisplotwasgeneratedwithd=256forvariouslevelsof sparsity s. The horizontal axis represents the number of measurements m, and theverticalaxisrepresentstheexactrecoverypercentage.Figure3.2.3depictsaplotofthevaluesformandsatwhich99%ofsparseatsignalsarerecoveredexactly. Thisplotwasgeneratedwithd = 256. Thehorizontalaxisrepresents thenumber of measurements m, andthevertical axisthesparsitylevels.Our resultsguarantee that CoSaMP reconstructssignals correctlywith justO(s)iterations. Figure 3.2.4depicts the number of iterations neededbyCoSaMPford = 10, 000 and m = 200 for perfect reconstruction. CoSaMP was executed under thesamesettingasdescribedabovefor sparseatsignalsaswellassparsecompressiblesignals for various values of p, andthenumber of iterations ineachscenariowasaveragedoverthe500trials. Theseaverageswereplottedagainstthesparsityofthesignal. Theplotdemonstratesthatoftenfarfeweriterationsareactuallyneededinsomecases. Thisisnotsurprising, sinceaswediscussedabovealternativehalting3.2.CompressiveSamplingMatchingPursuit 114criteriamaybe better inpractice. The plot alsodemonstrates that the numberof iterationsneededforsparsecompressibleishigher thanthenumber neededforsparseatsignals,asonewouldexpect. Theplotsuggeststhatforsmallervaluesofp(meaningsignalsthatdecaymorerapidly)CoSaMPneedsmoreiterations.0 50 100 150 200 250050100150200250300350400450500Number of measurements (m)Percentage recovered)Percentage of input signals recovered exactly (d=256) (Gaussian) s = 4s = 12s = 20s = 28s = 36Figure3.2.2: Thepercentageofsparseatsignals