Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity...
Transcript of Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity...
![Page 1: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/1.jpg)
Algorithms for sparse analysisLecture I: Background on sparse approximation
Anna C. Gilbert
Department of MathematicsUniversity of Michigan
![Page 2: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/2.jpg)
Tutorial on sparse approximations and algorithms
• Compress data
accuratelyconciselyefficiently (encoding and decoding)
• Focus on
mathematical and algorithmic theory: sparse approximationand compressive sensingalgorithms: polynomial time, randomized, sublinear,approximationbridge amongst engineering applications, mathematics, andalgorithms
• Not focus on
applicationsimage reconstruction or inverse problemsimage models, codecsdictionary design
![Page 3: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/3.jpg)
Lectures
• Lecture 1: Background, problem formulation
• Lecture 2: Computational complexity
• Lecture 3: Geometry of dictionaries, greedy algorithms,convex relaxation
• Lecture 4: Sublinear (approximation) algorithms
![Page 4: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/4.jpg)
Basic image compression: transform coding
![Page 5: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/5.jpg)
Orthogonal basis Φ: Transform coding
θ(c)
Φᵀ x = c
Φ ≈ x
• Compute orthogonal transform
ΦT x = c
• Threshold small coefficients
Θ(c)
• Reconstruct approximate image
Φ(Θ(c)) ≈ x
![Page 6: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/6.jpg)
Orthogonal basis Φ: Transform coding
θ(c)
Φᵀ x = c
Φ ≈ x
• Compute orthogonal transform
ΦT x = c
• Threshold small coefficients
Θ(c)
• Reconstruct approximate image
Φ(Θ(c)) ≈ x
![Page 7: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/7.jpg)
Orthogonal basis Φ: Transform coding
θ(c)
Φᵀ x = c
Φ ≈ x
• Compute orthogonal transform
ΦT x = c
• Threshold small coefficients
Θ(c)
• Reconstruct approximate image
Φ(Θ(c)) ≈ x
![Page 8: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/8.jpg)
Nonlinear encoding
=c xΦᵀ ⇒ x =Ψᵀ
a
c'
=c yΦᵀ ⇒ y =Ωᵀ c'
• Position of nonzeros depends on signal• Different matrices ΦT , ΩT for 2 different signals• Adaptive procedure, adapt for each input
![Page 9: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/9.jpg)
Nonlinear encoding
=c xΦᵀ ⇒ x =Ψᵀ
a
c'
=c yΦᵀ ⇒ y =Ωᵀ c'
• Position of nonzeros depends on signal• Different matrices ΦT , ΩT for 2 different signals• Adaptive procedure, adapt for each input
![Page 10: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/10.jpg)
Linear decoding
≈c xΦ
cΦ ≈ y
• Given the vector of coefficients andthe nonzero positions, recover(approximate) signal via linearcombination of coefficients andbasis vectors
Φ(Θ(c)) ≈ x
• Decoding procedure not signaldependent
• Matrix Φ same for all signals
![Page 11: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/11.jpg)
Linear decoding
≈c xΦ
cΦ ≈ y
• Given the vector of coefficients andthe nonzero positions, recover(approximate) signal via linearcombination of coefficients andbasis vectors
Φ(Θ(c)) ≈ x
• Decoding procedure not signaldependent
• Matrix Φ same for all signals
![Page 12: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/12.jpg)
Linear decoding
≈c xΦ
cΦ ≈ y
• Given the vector of coefficients andthe nonzero positions, recover(approximate) signal via linearcombination of coefficients andbasis vectors
Φ(Θ(c)) ≈ x
• Decoding procedure not signaldependent
• Matrix Φ same for all signals
![Page 13: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/13.jpg)
Nonlinear encoding/linear decoding
• Nonlinear encoding• Instance optimal: given
signal x , find best set ofcoefficients for given signal
• Algorithm does not guaranteebasis is a good one, onlyrepresentation good wrt basis
• Relatively easy to compute• But compressibility, sparsity of
signal depends on choice oforthonormal basis
• Hard to design bestorthonormal basis
• Linear decoding: easy tocompute
x =c
≈c xΦ
Φt
![Page 14: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/14.jpg)
Sparsity/Compressibility
• Sparsity: `0 “norm” = number of non-zero coefficients
• Compressibility: rate of decay of sorted coefficients
• Reflects sparsity/compressibility of signal in particular basis
• Fewer non-zero coefficients =⇒ fewer items to encode =⇒fewer total bits to represent signal
• Occam’s razor: capture essential features of signal
![Page 15: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/15.jpg)
Basic compressibility results
• DefinitionThe optimal k-term error for x ∈ RN
σk(x)p = minz k-sparse
‖x − z‖p.
The `p-error of best k-term representation for x in canonical basis.
• How to relate ‖x‖p to σk(x)p? restrict 1 ≤ p <∞.
• Proposition
If 1 ≤ p <∞ and q = (r + 1p )−1, then for all x ∈ RN ,
σk(x)p ≤ k−r‖x‖q.
![Page 16: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/16.jpg)
Basic compressibility results
• DefinitionThe optimal k-term error over the class K ⊂ RN
σk(K )p = supx∈K
σk(x)p.
• Let K = BNq = x ∈ RN | ‖x‖q = 1.
σk(K )p ≤ k−r
with r = 1q −
1p .
![Page 17: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/17.jpg)
Redundancy
If one orthonormal basis is good, surely two (or more) are better...
...especially for images
![Page 18: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/18.jpg)
Redundancy
If one orthonormal basis is good, surely two (or more) are better...
...especially for images
![Page 19: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/19.jpg)
![Page 20: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/20.jpg)
Dictionary
DefinitionA dictionary Φ in Rd is a collection ϕ`N`=1 ⊂ Rd of unit-normvectors: ‖ϕ`‖2 = 1 for all `.
• Elements are called atoms
• If spanϕ` = Rd , the dictionary is complete
• If ϕ` are linearly dependent, the dictionary is redundant
![Page 21: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/21.jpg)
Matrix representation
Form a matrixΦ =
[ϕ1 ϕ2 . . . ϕN
]so that
Φc =∑`
c`ϕ`.
x
Φᵀ
Φ c
![Page 22: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/22.jpg)
Examples: Fourier—Dirac
Φ = [F | I ]
ϕ`(t) =1√de2πi`t/d ` = 1, 2, . . . , d
ϕ`(t) = δ`(t) ` = d + 1, d + 2, . . . , 2d
![Page 23: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/23.jpg)
Examples: DCT—Wavelets, 2 dimensions
Φ = [F |W ]
![Page 24: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/24.jpg)
Examples: Wavelet packets
![Page 25: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/25.jpg)
Sparsity: Tradeoff
xΦ c =
cost of representation ←→ error in approximation
rate ←→ distortion
![Page 26: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/26.jpg)
Sparse Problems
• Exact. Given a vector x ∈ Rd and a complete dictionary Φ,solve
arg minc‖c‖0 s.t. x = Φc
i.e., find a sparsest representation of x over Φ.
• Error. Given ε ≥ 0, solve
arg minc‖c‖0 s.t. ‖x − Φc‖2 ≤ ε
i.e., find a sparsest approximation of x that achieves error ε.
• Sparse. Given k ≥ 1, solve
arg minc‖x − Φc‖2 s.t. ‖c‖0 ≤ k
i.e., find the best approximation of x using k atoms.
![Page 27: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/27.jpg)
Sparse Problems
• Exact. Given a vector x ∈ Rd and a complete dictionary Φ,solve
arg minc‖c‖0 s.t. x = Φc
i.e., find a sparsest representation of x over Φ.
• Error. Given ε ≥ 0, solve
arg minc‖c‖0 s.t. ‖x − Φc‖2 ≤ ε
i.e., find a sparsest approximation of x that achieves error ε.
• Sparse. Given k ≥ 1, solve
arg minc‖x − Φc‖2 s.t. ‖c‖0 ≤ k
i.e., find the best approximation of x using k atoms.
![Page 28: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/28.jpg)
Sparse Problems
• Exact. Given a vector x ∈ Rd and a complete dictionary Φ,solve
arg minc‖c‖0 s.t. x = Φc
i.e., find a sparsest representation of x over Φ.
• Error. Given ε ≥ 0, solve
arg minc‖c‖0 s.t. ‖x − Φc‖2 ≤ ε
i.e., find a sparsest approximation of x that achieves error ε.
• Sparse. Given k ≥ 1, solve
arg minc‖x − Φc‖2 s.t. ‖c‖0 ≤ k
i.e., find the best approximation of x using k atoms.
![Page 29: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/29.jpg)
Computational complexity
• How hard is it to solve these problems?
• How difficult are these problems as compared to well-studiedproblems; e.g., Primes, SAT?
• What is the asymptotic complexity of the problem?
• Is there a feasible algorithm that is guaranteed to solve theproblem?
• Sparse approximation problems are at least as hard as SAT
![Page 30: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/30.jpg)
Complexity theory: Decision problems
• Fundamental problem type: output is Yes or No
• RelPrime. Are a and b relatively prime?
• SAT. Boolean expression . Given a Boolean expression withand, or, not, variables, and parentheses only, is there someassignment of True and False to variables that makesexpression True?
• D-Exact. Given a vector x ∈ Rd , a complete d × Ndictionary Φ, and a sparsity parameter k , does there exist avector c ∈ RN with ‖c‖0 ≤ k such that Φc = x?
• Can be used as a subroutine (partially) to solve Exact
![Page 31: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/31.jpg)
Complexity theory: Languages
• Encode input as a finite string of 0s and 1s
• DefinitionA language L is a subset of the set of all (finite length) strings.
• Decision problem = language L
• Given input x , decide if x ∈ L or if x /∈ L
RelPrime = binary encodings of pairs (a,b) s.t. gcd(a, b) = 1
• Algorithm AL for deciding L
AL(x) =
Yes iff x ∈ L
No otherwise
![Page 32: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/32.jpg)
Complexity theory: P
• P = polynomial time
• DefinitionP is the class of languages L that are decidable in polynomial time;i.e., there is an algorithm AL such that
• x ∈ L iff AL(x) = Yes (otherwise AL(x) = No)• there is some n so that for all inputs x, the running time of AL
on x is less than |x |n
![Page 33: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/33.jpg)
Complexity theory: NP
NP 6= “Not Polynomial time”
![Page 34: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/34.jpg)
Complexity theory: Verify vs. determine
• Verifying existence of k-sparse vector c with Φc = x easierthan determining existence
• Given witness c , check (i) ‖c‖0 = k and (ii) Φc = x
• Checks can be done in time polynomial in size of Φ, x , k
• DefinitionA language L has a (polynomial time) verifier if there is analgorithm V such that
L = x | ∃w s.t. V accepts (x ,w)
(and algorithm V runs in time polynomial in the length of x).
![Page 35: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/35.jpg)
NP
• NP = nondeterministicpolynomial time
DefinitionNP is the class of languages Lthat have polynomial timeverifiers.
• P = membership decidedefficiently
• NP = membershipverified efficiently
• P = NP?
P = NP
NP
P
![Page 36: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/36.jpg)
NP
• NP = nondeterministicpolynomial time
DefinitionNP is the class of languages Lthat have polynomial timeverifiers.
• P = membership decidedefficiently
• NP = membershipverified efficiently
• P = NP?
P = NP
NP
P
![Page 37: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/37.jpg)
NP
• NP = nondeterministicpolynomial time
DefinitionNP is the class of languages Lthat have polynomial timeverifiers.
• P = membership decidedefficiently
• NP = membershipverified efficiently
• P = NP?
P = NP
NP
P
![Page 38: Algorithms for sparse analysis Lecture I: Background on sparse … · 2011-05-24 · Complexity theory: Verify vs. determine Verifying existence of k-sparse vector cwith = x easier](https://reader035.fdocuments.in/reader035/viewer/2022063002/5f48b4e96d5cac6438372c9e/html5/thumbnails/38.jpg)
Summary
• Formal, precise definitions of sparse approximation problems
• Set up complexity theory: goals, definitions, 2 complexityclasses
• Next lecture: reductions amongst problem, hardness results