Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar...
Transcript of Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar...
![Page 1: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/1.jpg)
Non-convex Robust PCA:Provable Bounds
Anima Anandkumar
U.C. Irvine
Joint work with Praneeth Netrapalli, U.N. Niranjan,Prateek Jain and Sujay Sanghavi.
![Page 2: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/2.jpg)
Learning with Big Data
High Dimensional Regime
Missing observations, gross corruptions, outliers, ill-posed problems.
Needle in a haystack: finding low dimensional structures in highdimensional data.
Principled approaches for finding low dimensional structures?
![Page 3: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/3.jpg)
PCA: Classical Method
Denoising: find hidden low rankstructures in data.
Efficient computation, perturbationanalysis.
![Page 4: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/4.jpg)
PCA: Classical Method
Denoising: find hidden low rankstructures in data.
Efficient computation, perturbationanalysis.
![Page 5: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/5.jpg)
PCA: Classical Method
Denoising: find hidden low rankstructures in data.
Efficient computation, perturbationanalysis.
Not robust to even a few outliers
![Page 6: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/6.jpg)
Robust PCA Problem
Find low rank structure after removing sparse corruptions.
Decompose input matrix as low rank + sparse matrices.
M L∗ S∗
M ∈ Rn×n, L∗ is low rank and S∗ is sparse.
Applications in computer vision, topic and community modeling.
![Page 7: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/7.jpg)
History
Heuristics without guarantes
Multivariate trimming [Gnanadeskian+ Kettering 72]
Random sampling [Fischler+ Bolles81].
Alternating minimization [Ke+ Kanade03].
Influence functions [de la Torre + Black 03]
Convex methods with Guarantees
Chandrasekharan et. al, Candes et. al ‘11: seminal guarantees.
Hsu et. al ‘11, Agarwal et. al ‘12: further guarantees.
(Variants) Xu et. al ‘11: Outlier pursuit, Chen et. al ‘12: communitydetection.
![Page 8: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/8.jpg)
Why is Robust PCA difficult?
M L∗ S∗
No identifiability in general: Low rank matrices can also be sparse andvice versa.
Natural constraints for identifiability?
Low rank matrix is NOT sparse and viceversa.
Incoherent low rank matrix and sparse matrix with sparsityconstraints.
Tractable methods for identifiable settings?
![Page 9: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/9.jpg)
Why is Robust PCA difficult?
M L∗ S∗
No identifiability in general: Low rank matrices can also be sparse andvice versa.
Natural constraints for identifiability?
Low rank matrix is NOT sparse and viceversa.
Incoherent low rank matrix and sparse matrix with sparsityconstraints.
Tractable methods for identifiable settings?
![Page 10: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/10.jpg)
Convex Relaxation Techniques
(Hard) Optimization Problem, given M ∈ Rn×n
minL,S
Rank(L) + γ‖S‖0, M = L+ S.
Rank(L) = {#σi(L) : σi(L) 6= 0}, ‖S‖0 = {#S(i, j) : S(i, j) 6= 0}are not tractable.
![Page 11: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/11.jpg)
Convex Relaxation Techniques
(Hard) Optimization Problem, given M ∈ Rn×n
minL,S
Rank(L) + γ‖S‖0, M = L+ S.
Rank(L) = {#σi(L) : σi(L) 6= 0}, ‖S‖0 = {#S(i, j) : S(i, j) 6= 0}are not tractable.
Convex Relaxation
minL,S‖L‖∗ + γ‖S‖1, M = L+ S.
‖L‖∗ =∑
i σi(L), ‖S‖1 =∑
i,j |S(i, j)| are convex sets.
Chandrasekharan et. al, Candes et. al ‘11: seminal works.
![Page 12: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/12.jpg)
Other Alternatives for Robust PCA?
minL,S‖L‖∗ + γ‖S‖1, M = L+ S.
Shortcomings of convex methods
![Page 13: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/13.jpg)
Other Alternatives for Robust PCA?
minL,S‖L‖∗ + γ‖S‖1, M = L+ S.
Shortcomings of convex methods
Computational cost: O(n3/ǫ) to achieve error of ǫ
◮ Requires SVD of n× n matrix.
Analysis: requires dual witness style arguments.
Conditions for success usually opaque.
![Page 14: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/14.jpg)
Other Alternatives for Robust PCA?
minL,S‖L‖∗ + γ‖S‖1, M = L+ S.
Shortcomings of convex methods
Computational cost: O(n3/ǫ) to achieve error of ǫ
◮ Requires SVD of n× n matrix.
Analysis: requires dual witness style arguments.
Conditions for success usually opaque.
Non-convex alternatives?
![Page 15: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/15.jpg)
Proposal for Non-convex Robust PCA
minL,S‖S‖0, s.t. M = L+ S, Rank(L) = r
![Page 16: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/16.jpg)
Proposal for Non-convex Robust PCA
minL,S‖S‖0, s.t. M = L+ S, Rank(L) = r
A non-convex heuristic (AltProj)
Initialize L,S = 0 and iterate:
L← Pr(M − S) and S ← Hζ(M − L) .
Pr(·): rank-r projection. Hζ(·): thresholding with ζ.
Computationally efficient: each operation is just a rank-r SVD orthresholding.
![Page 17: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/17.jpg)
Proposal for Non-convex Robust PCA
minL,S‖S‖0, s.t. M = L+ S, Rank(L) = r
A non-convex heuristic (AltProj)
Initialize L,S = 0 and iterate:
L← Pr(M − S) and S ← Hζ(M − L) .
Pr(·): rank-r projection. Hζ(·): thresholding with ζ.
Computationally efficient: each operation is just a rank-r SVD orthresholding.
Any hope for proving guarantees?
![Page 18: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/18.jpg)
Observations regarding non-convex analysis
Challenges
Multiple stable points: bad local optima, solution depends oninitialization.
Method may have very slow convergence or may not converge at all!
![Page 19: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/19.jpg)
Observations regarding non-convex analysis
Challenges
Multiple stable points: bad local optima, solution depends oninitialization.
Method may have very slow convergence or may not converge at all!
Non-convex Projections vs. Convex Projections
Projections on to non-convex sets: NP-hard in general.
◮ Projections on to rank and sparse sets: tractable.
Less information than convex projections: zero-order conditions.
‖P (M)−M‖ ≤ ‖Y −M‖, ∀Y ∈ C(Non-convex),
‖P (M) −M‖2 ≤ 〈Y −M,P (M) −M〉, ∀Y ∈ C(Convex).
![Page 20: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/20.jpg)
Non-convex success stories
Classical Result
PCA: Convergence to global optima!
![Page 21: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/21.jpg)
Non-convex success stories
Classical Result
PCA: Convergence to global optima!
Recent results
Tensor methods (Anandkumar et. al ‘12, ‘14): Local optima can becharacterized in special cases.
![Page 22: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/22.jpg)
Non-convex success stories
Classical Result
PCA: Convergence to global optima!
Recent results
Tensor methods (Anandkumar et. al ‘12, ‘14): Local optima can becharacterized in special cases.
Dictionary learning (Agarwal et. al ‘14, Arora et. al ‘14): Initializeusing a “clustering style” method and do alternating minimization.
![Page 23: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/23.jpg)
Non-convex success stories
Classical Result
PCA: Convergence to global optima!
Recent results
Tensor methods (Anandkumar et. al ‘12, ‘14): Local optima can becharacterized in special cases.
Dictionary learning (Agarwal et. al ‘14, Arora et. al ‘14): Initializeusing a “clustering style” method and do alternating minimization.
Matrix completion/phase retrieval: (Netrapalli et. al ‘13) Initializewith PCA and do alternating minimization.
![Page 24: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/24.jpg)
Non-convex success stories
Classical Result
PCA: Convergence to global optima!
Recent results
Tensor methods (Anandkumar et. al ‘12, ‘14): Local optima can becharacterized in special cases.
Dictionary learning (Agarwal et. al ‘14, Arora et. al ‘14): Initializeusing a “clustering style” method and do alternating minimization.
Matrix completion/phase retrieval: (Netrapalli et. al ‘13) Initializewith PCA and do alternating minimization.
(Somewhat) common theme
Characterize basin of attraction for global optimum.
Obtain a good initialization to “land in the ball”.
![Page 25: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/25.jpg)
Non-convex Robust PCA
A non-convex heuristic (AltProj)
Initialize L,S = 0 and iterate:
L← Pr(M − S) and S ← Hζ(M − L) .
Observations regarding Robust PCA
Projection on to rank and sparse subspaces: non-convex buttractable: SVD and hard thresholding.
But alternating projections: challenging to analyze
![Page 26: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/26.jpg)
Non-convex Robust PCA
A non-convex heuristic (AltProj)
Initialize L,S = 0 and iterate:
L← Pr(M − S) and S ← Hζ(M − L) .
Observations regarding Robust PCA
Projection on to rank and sparse subspaces: non-convex buttractable: SVD and hard thresholding.
But alternating projections: challenging to analyze
Our results for (a variant of) AltProj
Guaranteed recovery of low rank L∗ and sparse part S∗.
Match the bounds for convex methods (deterministic sparsity).
Reduced computation: only require low rank SVDs!
![Page 27: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/27.jpg)
Non-convex Robust PCA
A non-convex heuristic (AltProj)
Initialize L,S = 0 and iterate:
L← Pr(M − S) and S ← Hζ(M − L) .
Observations regarding Robust PCA
Projection on to rank and sparse subspaces: non-convex buttractable: SVD and hard thresholding.
But alternating projections: challenging to analyze
Our results for (a variant of) AltProj
Guaranteed recovery of low rank L∗ and sparse part S∗.
Match the bounds for convex methods (deterministic sparsity).
Reduced computation: only require low rank SVDs!
Best of both worlds: reduced computation with guarantees!
![Page 28: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/28.jpg)
Outline
1 Introduction
2 Analysis
3 Experiments
4 Robust Tensor PCA
5 Conclusion
![Page 29: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/29.jpg)
Toy example: Rank-1 case
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L) .
P1(·): rank-1 projection. Hζ(·): thresholding.
![Page 30: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/30.jpg)
Toy example: Rank-1 case
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L) .
P1(·): rank-1 projection. Hζ(·): thresholding.
Immediate Observations
First PCA: L← P1(M).
![Page 31: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/31.jpg)
Toy example: Rank-1 case
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L) .
P1(·): rank-1 projection. Hζ(·): thresholding.
Immediate Observations
First PCA: L← P1(M).
Matrix perturbation bound: ‖M − L‖2 ≤ O(‖S∗‖)
![Page 32: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/32.jpg)
Toy example: Rank-1 case
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L) .
P1(·): rank-1 projection. Hζ(·): thresholding.
Immediate Observations
First PCA: L← P1(M).
Matrix perturbation bound: ‖M − L‖2 ≤ O(‖S∗‖)If ‖S∗‖ ≫ 1, no progress!
![Page 33: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/33.jpg)
Toy example: Rank-1 case
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L) .
P1(·): rank-1 projection. Hζ(·): thresholding.
Immediate Observations
First PCA: L← P1(M).
Matrix perturbation bound: ‖M − L‖2 ≤ O(‖S∗‖)If ‖S∗‖ ≫ 1, no progress!
Exploit incoherence of L∗?
![Page 34: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/34.jpg)
Rank-1 Analysis Contd.
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L).
![Page 35: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/35.jpg)
Rank-1 Analysis Contd.
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L).
Incoherence of L∗
L∗ = u∗(u∗)⊤ and ‖u∗‖∞ ≤µ√n
and ‖L∗‖∞ ≤µ2
n.
![Page 36: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/36.jpg)
Rank-1 Analysis Contd.
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L).
Incoherence of L∗
L∗ = u∗(u∗)⊤ and ‖u∗‖∞ ≤µ√n
and ‖L∗‖∞ ≤µ2
n.
Solution for handling large ‖S∗‖First threshold M before rank-1 projection.
Ensures large entries of S∗ are identified.
![Page 37: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/37.jpg)
Rank-1 Analysis Contd.
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L,S = 0 and iterate:
L← P1(M − S) and S ← Hζ(M − L).
Incoherence of L∗
L∗ = u∗(u∗)⊤ and ‖u∗‖∞ ≤µ√n
and ‖L∗‖∞ ≤µ2
n.
Solution for handling large ‖S∗‖First threshold M before rank-1 projection.
Ensures large entries of S∗ are identified.
Choose threshold ζ0 =4µ2
n.
![Page 38: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/38.jpg)
Rank-1 Analysis Contd.
M = L∗ + S∗, L∗ = u∗(u∗)⊤
Non-convex method (AltProj)
Initialize L = 0, S = Hζ0(M) and iterate:
L← P1(M − S) and S ← Hζ(M − L).
Incoherence of L∗
L∗ = u∗(u∗)⊤ and ‖u∗‖∞ ≤µ√n
and ‖L∗‖∞ ≤µ2
n.
Solution for handling large ‖S∗‖First threshold M before rank-1 projection.
Ensures large entries of S∗ are identified.
Choose threshold ζ0 =4µ2
n.
![Page 39: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/39.jpg)
Rank-1 Analysis Contd.
Non-convex method (AltProj)
L(0) = 0, S(0) = Hζ0(M),
L(t+1) ← P1(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
M
L∗S∗
L(1)
S(0)
P1
Hζ1
Hζ2
To analyze progress, track E(t+1) := S∗ − S(t+1)
![Page 40: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/40.jpg)
Rank-1 Analysis Contd.
One iteration of AltProj
L(0) = 0, S(0) = Hζ0(M), L(1) ← P1(M − S(0)), S(1) ← Hζ(M − L(1)) .
Analyze E(1) := S∗ − S(1)
Thresholding is element-wise operation: require ‖L(1) − L∗‖∞.
In general, no special bound for ‖L(1) − L∗‖∞.
Exploit sparsity of S∗ and incoherence of L∗?
![Page 41: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/41.jpg)
Rank-1 Analysis Contd.L(1) = uu⊤ = P1(M − S(0)) and E(0) = S∗ − S(0).
Fixed point equation for eigenvectors (M − S(0))u = λu
![Page 42: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/42.jpg)
Rank-1 Analysis Contd.L(1) = uu⊤ = P1(M − S(0)) and E(0) = S∗ − S(0).
Fixed point equation for eigenvectors (M − S(0))u = λu
〈u∗, u〉u∗ + (S∗ − S(0))u = λu or u = λ〈u∗, u〉(
I − E(0)
λ
)−1u∗
Taylor Series
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
![Page 43: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/43.jpg)
Rank-1 Analysis Contd.L(1) = uu⊤ = P1(M − S(0)) and E(0) = S∗ − S(0).
Fixed point equation for eigenvectors (M − S(0))u = λu
〈u∗, u〉u∗ + (S∗ − S(0))u = λu or u = λ〈u∗, u〉(
I − E(0)
λ
)−1u∗
Taylor Series
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse: supp(E(0)) ⊆ supp(S∗).
![Page 44: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/44.jpg)
Rank-1 Analysis Contd.L(1) = uu⊤ = P1(M − S(0)) and E(0) = S∗ − S(0).
Fixed point equation for eigenvectors (M − S(0))u = λu
〈u∗, u〉u∗ + (S∗ − S(0))u = λu or u = λ〈u∗, u〉(
I − E(0)
λ
)−1u∗
Taylor Series
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse: supp(E(0)) ⊆ supp(S∗).
Exploiting sparsity: (E(0))p is the pth-hop adjacency matrix of E(0).
![Page 45: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/45.jpg)
Rank-1 Analysis Contd.L(1) = uu⊤ = P1(M − S(0)) and E(0) = S∗ − S(0).
Fixed point equation for eigenvectors (M − S(0))u = λu
〈u∗, u〉u∗ + (S∗ − S(0))u = λu or u = λ〈u∗, u〉(
I − E(0)
λ
)−1u∗
Taylor Series
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse: supp(E(0)) ⊆ supp(S∗).
Exploiting sparsity: (E(0))p is the pth-hop adjacency matrix of E(0).
Counting walks in sparse graphs.
![Page 46: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/46.jpg)
Rank-1 Analysis Contd.L(1) = uu⊤ = P1(M − S(0)) and E(0) = S∗ − S(0).
Fixed point equation for eigenvectors (M − S(0))u = λu
〈u∗, u〉u∗ + (S∗ − S(0))u = λu or u = λ〈u∗, u〉(
I − E(0)
λ
)−1u∗
Taylor Series
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse: supp(E(0)) ⊆ supp(S∗).
Exploiting sparsity: (E(0))p is the pth-hop adjacency matrix of E(0).
Counting walks in sparse graphs.
In addition, u∗ is incoherent: ‖u∗‖∞ <µ√n.
![Page 47: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/47.jpg)
Rank-1 Analysis Contd.
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse (each row/column is d sparse) and u∗ is µ-incoherent.
![Page 48: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/48.jpg)
Rank-1 Analysis Contd.
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse (each row/column is d sparse) and u∗ is µ-incoherent.
We show: ‖(E(0))pu∗‖∞ ≤µ√n(d‖E(0)‖∞)p .
![Page 49: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/49.jpg)
Rank-1 Analysis Contd.
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse (each row/column is d sparse) and u∗ is µ-incoherent.
We show: ‖(E(0))pu∗‖∞ ≤µ√n(d‖E(0)‖∞)p .
Convergence when terms are < 1, i.e. d‖E(0)‖∞ < 1.
![Page 50: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/50.jpg)
Rank-1 Analysis Contd.
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse (each row/column is d sparse) and u∗ is µ-incoherent.
We show: ‖(E(0))pu∗‖∞ ≤µ√n(d‖E(0)‖∞)p .
Convergence when terms are < 1, i.e. d‖E(0)‖∞ < 1.
Recall ‖E(0)‖∞ <4µ2
ndue to thresholding.
![Page 51: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/51.jpg)
Rank-1 Analysis Contd.
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse (each row/column is d sparse) and u∗ is µ-incoherent.
We show: ‖(E(0))pu∗‖∞ ≤µ√n(d‖E(0)‖∞)p .
Convergence when terms are < 1, i.e. d‖E(0)‖∞ < 1.
Recall ‖E(0)‖∞ <4µ2
ndue to thresholding.
Require d <n
4µ2. Can tolerate O(n) corruptions!
![Page 52: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/52.jpg)
Rank-1 Analysis Contd.
u = λ〈u∗, u〉
I +∑
p≥1
(
E(0)
λ
)p
u∗
E(0) is sparse (each row/column is d sparse) and u∗ is µ-incoherent.
We show: ‖(E(0))pu∗‖∞ ≤µ√n(d‖E(0)‖∞)p .
Convergence when terms are < 1, i.e. d‖E(0)‖∞ < 1.
Recall ‖E(0)‖∞ <4µ2
ndue to thresholding.
Require d <n
4µ2. Can tolerate O(n) corruptions!
Contraction of error E(t) when degree d is bounded.
![Page 53: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/53.jpg)
Extension to general rank: challenges
![Page 54: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/54.jpg)
Extension to general rank: challenges
A proposal for rank-r Non-convex method (AltProj)
Init L(0) = 0, S(0) = Hζ0(M), iterate:
L(t+1) ← Pr(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
![Page 55: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/55.jpg)
Extension to general rank: challenges
A proposal for rank-r Non-convex method (AltProj)
Init L(0) = 0, S(0) = Hζ0(M), iterate:
L(t+1) ← Pr(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
Recall for rank-1 case
Initial threshold controlled perturbation for rank-1 projection.
![Page 56: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/56.jpg)
Extension to general rank: challenges
A proposal for rank-r Non-convex method (AltProj)
Init L(0) = 0, S(0) = Hζ0(M), iterate:
L(t+1) ← Pr(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
Recall for rank-1 case
Initial threshold controlled perturbation for rank-1 projection.
Perturbation analysis in general rank case
Small λ∗min(L
∗): no recovery of lower eigenvectors.
![Page 57: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/57.jpg)
Extension to general rank: challenges
A proposal for rank-r Non-convex method (AltProj)
Init L(0) = 0, S(0) = Hζ0(M), iterate:
L(t+1) ← Pr(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
Recall for rank-1 case
Initial threshold controlled perturbation for rank-1 projection.
Perturbation analysis in general rank case
Small λ∗min(L
∗): no recovery of lower eigenvectors.
Sparsity level depends on condition number λ∗max/λ
∗min
![Page 58: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/58.jpg)
Extension to general rank: challenges
A proposal for rank-r Non-convex method (AltProj)
Init L(0) = 0, S(0) = Hζ0(M), iterate:
L(t+1) ← Pr(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
Recall for rank-1 case
Initial threshold controlled perturbation for rank-1 projection.
Perturbation analysis in general rank case
Small λ∗min(L
∗): no recovery of lower eigenvectors.
Sparsity level depends on condition number λ∗max/λ
∗min
Guarantees without dependence on condition number?
![Page 59: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/59.jpg)
Extension to general rank: challenges
A proposal for rank-r Non-convex method (AltProj)
Init L(0) = 0, S(0) = Hζ0(M), iterate:
L(t+1) ← Pr(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
Recall for rank-1 case
Initial threshold controlled perturbation for rank-1 projection.
Perturbation analysis in general rank case
Small λ∗min(L
∗): no recovery of lower eigenvectors.
Sparsity level depends on condition number λ∗max/λ
∗min
Guarantees without dependence on condition number?
Lower eigenvectors subject to a large perturbation initially.
![Page 60: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/60.jpg)
Extension to general rank: challenges
A proposal for rank-r Non-convex method (AltProj)
Init L(0) = 0, S(0) = Hζ0(M), iterate:
L(t+1) ← Pr(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
Recall for rank-1 case
Initial threshold controlled perturbation for rank-1 projection.
Perturbation analysis in general rank case
Small λ∗min(L
∗): no recovery of lower eigenvectors.
Sparsity level depends on condition number λ∗max/λ
∗min
Guarantees without dependence on condition number?
Lower eigenvectors subject to a large perturbation initially.
Reduce perturbation before recovering lower eigenvectors!
![Page 61: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/61.jpg)
Improved Algorithm for General Rank Setting
Stage-wise Projections
Init L(0) = 0, S(0) = Hζ0(M).
For stage k = 1 to r,
◮ Iterate: L(t+1) ← Pk(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
![Page 62: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/62.jpg)
Improved Algorithm for General Rank Setting
Stage-wise Projections
Init L(0) = 0, S(0) = Hζ0(M).
For stage k = 1 to r,
◮ Iterate: L(t+1) ← Pk(M − S(t)), S(t+1) ← Hζ(M − L(t+1)) .
M
L∗S∗
Pr
P2
L(1)
S(0)
P1
Hζ1
Hζ2
![Page 63: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/63.jpg)
Summary of Results
Low rank part: L∗ = U∗Λ∗(V ∗)⊤ has rank r.
Incoherence: ‖U∗(i, :)‖2, ‖V ∗(i, :)‖2 ≤µ√r√n.
Sparse part: S∗ has at most d non-zeros per row/column.
![Page 64: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/64.jpg)
Summary of Results
Low rank part: L∗ = U∗Λ∗(V ∗)⊤ has rank r.
Incoherence: ‖U∗(i, :)‖2, ‖V ∗(i, :)‖2 ≤µ√r√n.
Sparse part: S∗ has at most d non-zeros per row/column.
Theorem: Guarantees for Stage-wise AltProj
Exact recovery of L∗, S∗ when d = O
(
n
µ2r
)
![Page 65: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/65.jpg)
Summary of Results
Low rank part: L∗ = U∗Λ∗(V ∗)⊤ has rank r.
Incoherence: ‖U∗(i, :)‖2, ‖V ∗(i, :)‖2 ≤µ√r√n.
Sparse part: S∗ has at most d non-zeros per row/column.
Theorem: Guarantees for Stage-wise AltProj
Exact recovery of L∗, S∗ when d = O
(
n
µ2r
)
Computational complexity: O(
r2n2 log(1/ǫ))
![Page 66: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/66.jpg)
Summary of Results
Low rank part: L∗ = U∗Λ∗(V ∗)⊤ has rank r.
Incoherence: ‖U∗(i, :)‖2, ‖V ∗(i, :)‖2 ≤µ√r√n.
Sparse part: S∗ has at most d non-zeros per row/column.
Theorem: Guarantees for Stage-wise AltProj
Exact recovery of L∗, S∗ when d = O
(
n
µ2r
)
Computational complexity: O(
r2n2 log(1/ǫ))
Comparison to convex method
Same (deterministic) condition on d. Running time: O(
n3/ǫ)
![Page 67: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/67.jpg)
Summary of Results
Low rank part: L∗ = U∗Λ∗(V ∗)⊤ has rank r.
Incoherence: ‖U∗(i, :)‖2, ‖V ∗(i, :)‖2 ≤µ√r√n.
Sparse part: S∗ has at most d non-zeros per row/column.
Theorem: Guarantees for Stage-wise AltProj
Exact recovery of L∗, S∗ when d = O
(
n
µ2r
)
Computational complexity: O(
r2n2 log(1/ǫ))
Comparison to convex method
Same (deterministic) condition on d. Running time: O(
n3/ǫ)
Best of both worlds: reduced computation with guarantees!
“Non-convex Robust PCA,” P. Netrapalli, U.N. Niranjan, S. Sanghavi, A. , P. Jain, NIPS ‘14.
![Page 68: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/68.jpg)
Outline
1 Introduction
2 Analysis
3 Experiments
4 Robust Tensor PCA
5 Conclusion
![Page 69: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/69.jpg)
Synthetic Results
NcRPCA: Non-convex Robust PCA.
IALM: Inexact augmented Lagrange multipliers.
400 600 800
102
n α
Tim
e(s)
n = 2000, r = 5, µ = 1
NcRPCAIALM
1 1.5 2 2.5 3
102
µ
Tim
e(s)
n = 2000, r = 10, n α = 100
NcRPCAIALM
100 150 200
102
r
Tim
e(s)
n = 2000, n α = 3 r, µ = 1
NcRPCAIALM
100 150 200200
400
600
800
1000
r
Max
. Ran
k
n = 2000, n α = 3 r, µ = 1
IALM
1 1.5 2 2.5 3
200
300
400
500
600
µ
Max
. Ran
k
n = 2000, r = 10, n α = 100
IALM
2 4 6 8 10 12 1410
1
102
Iterations
Ran
k
n = 2000, r = 10, n α = 100
IALM
![Page 70: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/70.jpg)
Real data: Foreground/background Separation
Original Rank-10 PCA
AltProj IALM
![Page 71: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/71.jpg)
Real data: Foreground/background Separation
AltProj
IALM
![Page 72: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/72.jpg)
Outline
1 Introduction
2 Analysis
3 Experiments
4 Robust Tensor PCA
5 Conclusion
![Page 73: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/73.jpg)
Robust Tensor PCA
vs.
![Page 74: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/74.jpg)
Robust Tensor PCA
vs.
Robust Tensor Problem
![Page 75: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/75.jpg)
Robust Tensor PCA
vs.
Robust Tensor Problem
Applications: Robust Learning of Latent Variable Models.
A. , R. Ge, D. Hsu, S.M. Kakade and M. Telgarsky “Tensor Decompositions for Learning Latent
Variable Models,” Preprint, Oct. ‘12.
![Page 76: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/76.jpg)
Challenges and Preliminary Observations
T = L∗ + S∗ ∈ Rn×n×n, L∗ =
∑
i∈[r] a⊗3i .
Convex methods
No natural convex surrogate for tensor (CP) rank.
Matricization loses the tensor structure!
![Page 77: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/77.jpg)
Challenges and Preliminary Observations
T = L∗ + S∗ ∈ Rn×n×n, L∗ =
∑
i∈[r] a⊗3i .
Convex methods
No natural convex surrogate for tensor (CP) rank.
Matricization loses the tensor structure!
Non-Convex Heuristic: Extension of Matrix AltProj
L(t+1) ← Pr(T − S(t)), S(t+1) ← Hζ(T − L(t+1)) .
![Page 78: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/78.jpg)
Challenges and Preliminary Observations
T = L∗ + S∗ ∈ Rn×n×n, L∗ =
∑
i∈[r] a⊗3i .
Convex methods
No natural convex surrogate for tensor (CP) rank.
Matricization loses the tensor structure!
Non-Convex Heuristic: Extension of Matrix AltProj
L(t+1) ← Pr(T − S(t)), S(t+1) ← Hζ(T − L(t+1)) .
Challenges in Non-Convex Analysis
Pr for a general tensor is NP-hard!
![Page 79: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/79.jpg)
Challenges and Preliminary Observations
T = L∗ + S∗ ∈ Rn×n×n, L∗ =
∑
i∈[r] a⊗3i .
Convex methods
No natural convex surrogate for tensor (CP) rank.
Matricization loses the tensor structure!
Non-Convex Heuristic: Extension of Matrix AltProj
L(t+1) ← Pr(T − S(t)), S(t+1) ← Hζ(T − L(t+1)) .
Challenges in Non-Convex Analysis
Pr for a general tensor is NP-hard!
Can be well approximated in special cases, e.g. full rank factors.
![Page 80: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/80.jpg)
Challenges and Preliminary Observations
T = L∗ + S∗ ∈ Rn×n×n, L∗ =
∑
i∈[r] a⊗3i .
Convex methods
No natural convex surrogate for tensor (CP) rank.
Matricization loses the tensor structure!
Non-Convex Heuristic: Extension of Matrix AltProj
L(t+1) ← Pr(T − S(t)), S(t+1) ← Hζ(T − L(t+1)) .
Challenges in Non-Convex Analysis
Pr for a general tensor is NP-hard!
Can be well approximated in special cases, e.g. full rank factors.
Guaranteed recovery possible!
![Page 81: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/81.jpg)
Outline
1 Introduction
2 Analysis
3 Experiments
4 Robust Tensor PCA
5 Conclusion
![Page 82: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/82.jpg)
Conclusion
M L∗ S∗
Guaranteed Non-Convex Robust PCA
Simple non-convex method for robust PCA.
Alternating rank projections and thresholding.
Estimates for low rank and sparse parts “grown gradually”.
Guarantees match convex methods.
Low computational complexity: scalable to large matrices.
Possible to have both: guarantees and low computation!
![Page 83: Non-convex Robust PCA: Provable Bounds · Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and](https://reader033.fdocuments.in/reader033/viewer/2022041919/5e6b39821a10e9235347c932/html5/thumbnails/83.jpg)
Outlook
Reduce computational complexity? Skip stages in rank projections?Tight bounds for incoherent row-column subspaces?
Extendable to the tensor setting with tight scaling guarantees.
Other problems where non-convex methods have guarantees?
◮ Csiszar’s alternating minimization framework.
(Laserre) hierarchy for convex methods: increasing complexity for“harder” problems.
Analogous unified thinking for non-convex methods?
Holy grail: A general framework for non-convex methods?