Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital...
Transcript of Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital...
![Page 1: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/1.jpg)
29.10.2014
1
1/20
Department of Electrical EngineeringUniversity of Isfahan
Introduction to
Compressive
SensingSection I: Sparsity
2
References
Y. C. Eldar and G. Kutyniok, Compressed Sensing: Theory
and Application, Cambridge University Press, 2012.
Fundamental of Image and Video Processing Online course
![Page 2: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/2.jpg)
29.10.2014
2
3
Outlines
4
Introduction
Nyquist theory Vs. Compressive Sensing
DSP Reconstruction filter
Analog Sampling Kernels
DSP
Analog Interpolation
Kernels
![Page 3: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/3.jpg)
29.10.2014
3
5
Introduction
Nyquist theory Vs. Compressive Sensing
DSP Reconstruction filter
Analog Sampling Kernels
DSP
Analog Interpolation
Kernels
Nyquist Theory CS Theory
infinite-length, continuous-time Sig
Sampling Measuring
Simple recovery Special and nonlinear recovery
6
Introduction
Nyquist
sampling:
Compressive
Sampling:
Many beautiful papers covering theory, algorithms, and applications
b A x
b = Ax
b A x
=
=
![Page 4: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/4.jpg)
29.10.2014
4
7
Outlines
8
Review of Vector Spaces
𝒙 =𝟑𝟎𝟒
𝒙 2 = 3 2 + 4 2 = 25 = 5
𝒙 1 = 3 + 4 = 7
𝒙 0 = 2
𝒒𝒖𝒂𝒔𝒊𝒏𝒐𝒓𝒎
![Page 5: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/5.jpg)
29.10.2014
5
9
Review of Vector Spaces
𝒒𝒖𝒂𝒔𝒊𝒏𝒐𝒓𝒎
10
Review of Vector Spaces
![Page 6: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/6.jpg)
29.10.2014
6
11
Outlines
12
Signal Modeling
Low Dimensional Signal Models
Sparse ModelsLow-Rank Matrix Models
Parametric Models
Finite Union of Subspaces
Sparsity…
![Page 7: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/7.jpg)
29.10.2014
7
13
Outlines
14
What is Sparsity?
A vector is said to be sparse if it only has “a few” non-zero components.
The vector can represent a signal, witch my be sparse in its native domain (e.g., image of sky at night) or can be made sparse in another domain (e.g., natural images in the DCT domain)
A sparse vector may originate in numerous applications
![Page 8: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/8.jpg)
29.10.2014
8
15
What is Sparsity?
A vector is said to be sparse if it only has “a few” non-zero components.
The vector can represent a signal, witch my be sparse in its native domain (e.g., image of sky at night) or can be made sparse in another domain (e.g., natural images in the DCT domain)
A sparse vector may originate in numerous applications
=
16
Outlines
![Page 9: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/9.jpg)
29.10.2014
9
17
Applications
Compressive Sensing
Image and Video Processing
Machine learning
Statistics
Genetics
Econometrics
Neuroscience
…
18
Econometrics
sparse
![Page 10: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/10.jpg)
29.10.2014
10
19
Robust Regression
Least Square Method:
20
Robust Regression
Least Square Method:
![Page 11: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/11.jpg)
29.10.2014
11
21
Recommender Systems
Matrix Completion problem
Rank Minimization Problem
Low Rank Matrix Many of Singular Values are zero
22
Image Denoising
DCT Domain Bases Sparse
Smooth
• Image Inpating• Super Resolution Image• Face Rrecognition• Video Surveillance• …
![Page 12: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/12.jpg)
29.10.2014
12
23
Compressive Sensing
24
Outlines
![Page 13: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/13.jpg)
29.10.2014
13
25
Linear Inverse Problems
26
Linear Inverse Problems
![Page 14: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/14.jpg)
29.10.2014
14
27
Linear Inverse Problems
It depends on the application
Regularization Principle: Adding priori knowledge to problem
Sparse
28
Unit Sphere
![Page 15: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/15.jpg)
29.10.2014
15
29
30
It’s not a sparse solution
Derivation of closed form solution
![Page 16: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/16.jpg)
29.10.2014
16
31
Solutions are sparse
32
Solution is sparse
Basis Pursuit Problem
![Page 17: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/17.jpg)
29.10.2014
17
33
Solution is sparse
34
On Convexity
Non-convex functionConvex function
Convex set Non-convex set
![Page 18: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/18.jpg)
29.10.2014
18
35
Convex Optimization Problem
Convex function Convex set
36
A
![Page 19: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/19.jpg)
29.10.2014
19
37
Small solutionClosed form
Sparse solutionNon- convex
Sparse solutionconvex Sparse solution
NP-Hard
Greedy approaches (Matching Pursuit) approximate the solution
Can be solved via convex optimization algorithms
38
Outlines
![Page 20: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/20.jpg)
29.10.2014
20
39
Problem Reformulation
Noise in Observation
Swapping the Constraint and Objective
40
Problem Reformulation
Noise in Observation
Swapping the Constraint and Objective
convex convex
![Page 21: Exploiting Structure in Data - University of Isfahanengold.ui.ac.ir/~sabahi/Advanced digital signal... · 2016-05-08 · DSP Reconstruction filter Analog Sampling Kernels DSP Interpolation](https://reader033.fdocuments.in/reader033/viewer/2022060401/5f0e2ca07e708231d43df712/html5/thumbnails/21.jpg)
29.10.2014
21
41
Problem Reformulation
Bring constraint to objective
LASSO Problem (least absolute shrinkage and selection operator)