An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9....
Transcript of An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9....
![Page 1: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/1.jpg)
An Online Learning View via a Projections’ Path in theSparse-land
Sergios Theodoridis1
1Dept. of Informatics and Telecommunications, National and Kapodistrian Universityof Athens, Athens, Greece.
Workshop on Sparse Signal ProcessingFriday, Sep. 16, 2016
Joint work withP. Bouboulis, S. Chouvardas, Y. Kopsinis, G. Papageorgiou, K. Slavakis
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 1/58
![Page 2: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/2.jpg)
Sparsity
Sparse Modeling
• Sparse modeling has been a major focus of research effort overthe last decade or so.
• Sparsity promoting regularization of cost functions copes with:
Ill conditioning-overfitting when solving inverse problems; Learningfrom data is an instance of inverse problems.
Promote zeros when the underlying models have manynear-to-zero values.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 2/58
![Page 3: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/3.jpg)
Sparsity
Sparse Modeling
• Sparse modeling has been a major focus of research effort overthe last decade or so.
• Sparsity promoting regularization of cost functions copes with:
Ill conditioning-overfitting when solving inverse problems; Learningfrom data is an instance of inverse problems.
Promote zeros when the underlying models have manynear-to-zero values.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 2/58
![Page 4: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/4.jpg)
Sparsity
Sparse Modeling
• Sparse modeling has been a major focus of research effort overthe last decade or so.
• Sparsity promoting regularization of cost functions copes with:
Ill conditioning-overfitting when solving inverse problems; Learningfrom data is an instance of inverse problems.
Promote zeros when the underlying models have manynear-to-zero values.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 2/58
![Page 5: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/5.jpg)
Sparsity
Sparse Modeling
• Sparse modeling has been a major focus of research effort overthe last decade or so.
• Sparsity promoting regularization of cost functions copes with:
Ill conditioning-overfitting when solving inverse problems; Learningfrom data is an instance of inverse problems.
Promote zeros when the underlying models have manynear-to-zero values.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 2/58
![Page 6: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/6.jpg)
Sparse Modeling
The need for sparse Models: Two examples
• Compression
• Echo Cancelation
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 3/58
![Page 7: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/7.jpg)
Sparse Modeling
The Generic Model
OUTPUT=INPUT× SPARSE MODEL+NOISE
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 4/58
![Page 8: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/8.jpg)
Sparse Modeling
The Regression Model
• A generic model that covers a large class of problems (Filtering,Prediction)
yn = uTna∗ + vn
a∗ ∈ RL, is the unknown vector.un ∈ RL, is the incoming signal (sensing vectors).yn ∈ R, is the observed signal (measurements).vn is the additive noise process.
• a∗ is assumed to be sparse. That is, only a few, K << L, of itscomponents are nonzero
a∗ = [0, 0, ?︸︷︷︸1
, 0, . . . , 0, ?︸︷︷︸2
, 0, 0, . . . , 0, ?︸︷︷︸K
, 0, . . . , 0]T
• In its simplest formulation the task comprises the estimation ofa∗, based on a set of measurements (yn,un), n = 1 . . . N .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 5/58
![Page 9: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/9.jpg)
Sparse Modeling
The Regression Model
• A generic model that covers a large class of problems (Filtering,Prediction)
yn = uTna∗ + vn
a∗ ∈ RL, is the unknown vector.un ∈ RL, is the incoming signal (sensing vectors).yn ∈ R, is the observed signal (measurements).vn is the additive noise process.
• a∗ is assumed to be sparse. That is, only a few, K << L, of itscomponents are nonzero
a∗ = [0, 0, ?︸︷︷︸1
, 0, . . . , 0, ?︸︷︷︸2
, 0, 0, . . . , 0, ?︸︷︷︸K
, 0, . . . , 0]T
• In its simplest formulation the task comprises the estimation ofa∗, based on a set of measurements (yn,un), n = 1 . . . N .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 5/58
![Page 10: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/10.jpg)
Sparse Modeling
The Regression Model
• A generic model that covers a large class of problems (Filtering,Prediction)
yn = uTna∗ + vn
a∗ ∈ RL, is the unknown vector.un ∈ RL, is the incoming signal (sensing vectors).yn ∈ R, is the observed signal (measurements).vn is the additive noise process.
• a∗ is assumed to be sparse. That is, only a few, K << L, of itscomponents are nonzero
a∗ = [0, 0, ?︸︷︷︸1
, 0, . . . , 0, ?︸︷︷︸2
, 0, 0, . . . , 0, ?︸︷︷︸K
, 0, . . . , 0]T
• In its simplest formulation the task comprises the estimation ofa∗, based on a set of measurements (yn,un), n = 1 . . . N .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 5/58
![Page 11: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/11.jpg)
Sparse Modeling
The Regression Model
• A generic model that covers a large class of problems (Filtering,Prediction)
yn = uTna∗ + vn
a∗ ∈ RL, is the unknown vector.un ∈ RL, is the incoming signal (sensing vectors).yn ∈ R, is the observed signal (measurements).vn is the additive noise process.
• a∗ is assumed to be sparse. That is, only a few, K << L, of itscomponents are nonzero
a∗ = [0, 0, ?︸︷︷︸1
, 0, . . . , 0, ?︸︷︷︸2
, 0, 0, . . . , 0, ?︸︷︷︸K
, 0, . . . , 0]T
• In its simplest formulation the task comprises the estimation ofa∗, based on a set of measurements (yn,un), n = 1 . . . N .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 5/58
![Page 12: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/12.jpg)
Sparse Modeling
The Regression Model
• A generic model that covers a large class of problems (Filtering,Prediction)
yn = uTna∗ + vn
a∗ ∈ RL, is the unknown vector.un ∈ RL, is the incoming signal (sensing vectors).yn ∈ R, is the observed signal (measurements).vn is the additive noise process.
• a∗ is assumed to be sparse. That is, only a few, K << L, of itscomponents are nonzero
a∗ = [0, 0, ?︸︷︷︸1
, 0, . . . , 0, ?︸︷︷︸2
, 0, 0, . . . , 0, ?︸︷︷︸K
, 0, . . . , 0]T
• In its simplest formulation the task comprises the estimation ofa∗, based on a set of measurements (yn,un), n = 1 . . . N .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 5/58
![Page 13: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/13.jpg)
Sparse Modeling
The Regression Model
• A generic model that covers a large class of problems (Filtering,Prediction)
yn = uTna∗ + vn
a∗ ∈ RL, is the unknown vector.un ∈ RL, is the incoming signal (sensing vectors).yn ∈ R, is the observed signal (measurements).vn is the additive noise process.
• a∗ is assumed to be sparse. That is, only a few, K << L, of itscomponents are nonzero
a∗ = [0, 0, ?︸︷︷︸1
, 0, . . . , 0, ?︸︷︷︸2
, 0, 0, . . . , 0, ?︸︷︷︸K
, 0, . . . , 0]T
• In its simplest formulation the task comprises the estimation ofa∗, based on a set of measurements (yn,un), n = 1 . . . N .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 5/58
![Page 14: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/14.jpg)
Sparse Modeling
Dictionary Learning
• This is a powerful tool in analysing signals in terms ofovercomplete basis vectors.
[y1, . . . ,yN ]︸ ︷︷ ︸L×N
= [u1, . . . ,um]︸ ︷︷ ︸L×m
[a1, . . . ,aN ]︸ ︷︷ ︸m×N
, m>L
Y = UA
yn,∈ RL n = 1, 2, . . . , N , are the observation vectors.ui ∈ RL, i = 1, 2, . . . ,m, are the unknown atoms of thedictionary.an ∈ Rm, n = 1, 2, . . . , N , are the vectors of the unknownweights, corresponding in the respective expansion of the nthinput vector:
yn =
m∑i=1
uiani
where, an, n = 1, 2, . . . , N , sparse vectors.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 6/58
![Page 15: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/15.jpg)
Sparse Modeling
Dictionary Learning
• This is a powerful tool in analysing signals in terms ofovercomplete basis vectors.
[y1, . . . ,yN ]︸ ︷︷ ︸L×N
= [u1, . . . ,um]︸ ︷︷ ︸L×m
[a1, . . . ,aN ]︸ ︷︷ ︸m×N
, m>L
Y = UA
yn,∈ RL n = 1, 2, . . . , N , are the observation vectors.ui ∈ RL, i = 1, 2, . . . ,m, are the unknown atoms of thedictionary.an ∈ Rm, n = 1, 2, . . . , N , are the vectors of the unknownweights, corresponding in the respective expansion of the nthinput vector:
yn =
m∑i=1
uiani
where, an, n = 1, 2, . . . , N , sparse vectors.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 6/58
![Page 16: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/16.jpg)
Sparse Modeling
Dictionary Learning
• This is a powerful tool in analysing signals in terms ofovercomplete basis vectors.
[y1, . . . ,yN ]︸ ︷︷ ︸L×N
= [u1, . . . ,um]︸ ︷︷ ︸L×m
[a1, . . . ,aN ]︸ ︷︷ ︸m×N
, m>L
Y = UA
yn,∈ RL n = 1, 2, . . . , N , are the observation vectors.ui ∈ RL, i = 1, 2, . . . ,m, are the unknown atoms of thedictionary.an ∈ Rm, n = 1, 2, . . . , N , are the vectors of the unknownweights, corresponding in the respective expansion of the nthinput vector:
yn =
m∑i=1
uiani
where, an, n = 1, 2, . . . , N , sparse vectors.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 6/58
![Page 17: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/17.jpg)
Sparse Modeling
Low Rank Matrix Factorization
• This task is at the heart of dimensionality reduction.
Y = UA
=
r∑i=1
uiaTi
[y1, . . . ,yN ]︸ ︷︷ ︸L×N
= [u1, . . . ,ur]︸ ︷︷ ︸L×r
aT1...
arT
︸ ︷︷ ︸
r×N
• r < N .
• PCA performs low rank matrix factorization, by imposing sparsityon the singular values as well as orthogonality on U .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 7/58
![Page 18: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/18.jpg)
Sparse Modeling
Low Rank Matrix Factorization
• This task is at the heart of dimensionality reduction.
Y = UA
=
r∑i=1
uiaTi
[y1, . . . ,yN ]︸ ︷︷ ︸L×N
= [u1, . . . ,ur]︸ ︷︷ ︸L×r
aT1...
arT
︸ ︷︷ ︸
r×N
• r < N .
• PCA performs low rank matrix factorization, by imposing sparsityon the singular values as well as orthogonality on U .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 7/58
![Page 19: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/19.jpg)
Sparse Modeling
Low Rank Matrix Factorization
• Matrix Completion is a special constrained version of low rankmatrix factorization
• Y has missing elements and the lower rank matrix factorization isconstrained to provide the non-missing elements at the respectivepositions
Y =
∗ ∗ ∗ ∗ ∗ ∗∗ ∗ ∗ ∗ ∗ ∗...
......
......
...∗ ∗ ∗ ∗ ∗ ∗
=
r∑i=1
uiaTi
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 8/58
![Page 20: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/20.jpg)
Sparse Modeling
Low Rank Matrix Factorization
• Matrix Completion is a special constrained version of low rankmatrix factorization
• Y has missing elements and the lower rank matrix factorization isconstrained to provide the non-missing elements at the respectivepositions
Y =
∗ ∗ ∗ ∗ ∗ ∗∗ ∗ ∗ ∗ ∗ ∗...
......
......
...∗ ∗ ∗ ∗ ∗ ∗
=
r∑i=1
uiaTi
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 8/58
![Page 21: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/21.jpg)
Sparse Modeling
Low Rank Matrix Factorization
• Robust PCA is another special constrained version of low rankmatrix factorization.
Y = L+ V
L is a low rank matrix and V is a sparse matrix. The latter
models OUTLIER NOISE. Being outlier is sparse.
• The goal of the task is to obtain estimates L and V by imposingsparsity on the singular values of Y as well as on the elements ofV , constrained so that Y = L+ V .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 9/58
![Page 22: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/22.jpg)
Sparse Modeling
Low Rank Matrix Factorization
• Robust PCA is another special constrained version of low rankmatrix factorization.
Y = L+ V
L is a low rank matrix and V is a sparse matrix. The latter
models OUTLIER NOISE. Being outlier is sparse.
• The goal of the task is to obtain estimates L and V by imposingsparsity on the singular values of Y as well as on the elements ofV , constrained so that Y = L+ V .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 9/58
![Page 23: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/23.jpg)
Sparse Modeling
Low Rank Matrix Factorization
• Robust PCA is another special constrained version of low rankmatrix factorization.
Y = L+ V
L is a low rank matrix and V is a sparse matrix. The latter
models OUTLIER NOISE. Being outlier is sparse.
• The goal of the task is to obtain estimates L and V by imposingsparsity on the singular values of Y as well as on the elements ofV , constrained so that Y = L+ V .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 9/58
![Page 24: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/24.jpg)
Sparse Modeling
Robust Regression
• Robust Regression is an old problem, with a major impact comingfrom the works of Huber. The revival of interest is due to a newlook via sparsity-aware learning techniques. For example, thenoise may comprise a few large values (outliers) on top of theGaussian component. Since the large values are only a few, theycan be treated via sparse modeling arguments.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 10/58
![Page 25: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/25.jpg)
Sparse Regression Modeling
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch Learning Problem
Linear Regression Model yn = uTna∗ + vn
• U := [u1,u2, . . . ,uN ]T ∈ RN×L
• y := [y1, y2, . . . , yN ]T ∈ RN , and v := [v1, v2, . . . , vN ]T ∈ RN .
Batch Formulation: y = Ua∗ + v
=
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 11/58
![Page 26: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/26.jpg)
Sparse Regression Modeling
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch Learning Problem
Linear Regression Model yn = uTna∗ + vn
• U := [u1,u2, . . . ,uN ]T ∈ RN×L
• y := [y1, y2, . . . , yN ]T ∈ RN , and v := [v1, v2, . . . , vN ]T ∈ RN .
Batch Formulation: y = Ua∗ + v
=
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 11/58
![Page 27: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/27.jpg)
Sparse Regression Modeling
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch Learning Problem
Linear Regression Model yn = uTna∗ + vn
• U := [u1,u2, . . . ,uN ]T ∈ RN×L
• y := [y1, y2, . . . , yN ]T ∈ RN , and v := [v1, v2, . . . , vN ]T ∈ RN .
Batch Formulation: y = Ua∗ + v
=
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 11/58
![Page 28: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/28.jpg)
Estimating the unknown
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch vs Online Learning
Batch formulation: y = Ua∗ + v
Online Formulation: yn = uTna∗ + vn,
obtain an estimate, an, after (yn,un) has been received
========
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 12/58
![Page 29: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/29.jpg)
Estimating the unknown
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch vs Online Learning
Batch formulation: y = Ua∗ + v
Online Formulation: yn = uTna∗ + vn,
obtain an estimate, an, after (yn,un) has been received
========
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 12/58
![Page 30: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/30.jpg)
Estimating the unknown
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch vs Online Learning
Batch formulation: y = Ua∗ + v
Online Formulation: yn = uTna∗ + vn,
obtain an estimate, an, after (yn,un) has been received
========
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 12/58
![Page 31: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/31.jpg)
Estimating the unknown
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch vs Online Learning
Batch formulation: y = Ua∗ + v
Online Formulation: yn = uTna∗ + vn,
obtain an estimate, an, after (yn,un) has been received
=
=======
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 12/58
![Page 32: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/32.jpg)
Estimating the unknown
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch vs Online Learning
Batch formulation: y = Ua∗ + v
Online Formulation: yn = uTna∗ + vn,
obtain an estimate, an, after (yn,un) has been received
===
=====
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 12/58
![Page 33: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/33.jpg)
Estimating the unknown
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch vs Online Learning
Batch formulation: y = Ua∗ + v
Online Formulation: yn = uTna∗ + vn,
obtain an estimate, an, after (yn,un) has been received
======
==
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 12/58
![Page 34: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/34.jpg)
Estimating the unknown
There are two paths that lead to the “truth”, e.g, obtain an estimate aof the unknown a∗.
Batch vs Online Learning
Batch formulation: y = Ua∗ + v
Online Formulation: yn = uTna∗ + vn,
obtain an estimate, an, after (yn,un) has been received
========
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 12/58
![Page 35: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/35.jpg)
Sparse Vs Online Learning
Sparsity-promoting Batch algorithm(Compressed Sensing)
• Are mobilized after a finitenumber of data, (un, yn)
N−1n=0 , is
collected.
• For any new datum, theestimation of a∗, is repeatedfrom scratch.
• Computational complexity mightbecome prohibitive.
• Excessive storage demands.
• It is a “mature” research fieldwith a diverse number oftechniques and applications.
Sparsity-promoting Online algorithms
• Infinite number of data.
• For any new datum, the estimateof a∗ is updated dynamically.
• Cases of time-varying a∗ are“naturally” handled.
• Low complexity is required forstreaming applications.
• Fast convergence / Tracking.
• Large potential in Big Dataapplications
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 13/58
![Page 36: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/36.jpg)
Sparse Vs Online Learning
Sparsity-promoting Batch algorithm(Compressed Sensing)
• Are mobilized after a finitenumber of data, (un, yn)
N−1n=0 , is
collected.
• For any new datum, theestimation of a∗, is repeatedfrom scratch.
• Computational complexity mightbecome prohibitive.
• Excessive storage demands.
• It is a “mature” research fieldwith a diverse number oftechniques and applications.
Sparsity-promoting Online algorithms
• Infinite number of data.
• For any new datum, the estimateof a∗ is updated dynamically.
• Cases of time-varying a∗ are“naturally” handled.
• Low complexity is required forstreaming applications.
• Fast convergence / Tracking.
• Large potential in Big Dataapplications
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 13/58
![Page 37: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/37.jpg)
Sparse Vs Online Learning
Sparsity-promoting Batch algorithm(Compressed Sensing)
• Are mobilized after a finitenumber of data, (un, yn)
N−1n=0 , is
collected.
• For any new datum, theestimation of a∗, is repeatedfrom scratch.
• Computational complexity mightbecome prohibitive.
• Excessive storage demands.
• It is a “mature” research fieldwith a diverse number oftechniques and applications.
Sparsity-promoting Online algorithms
• Infinite number of data.
• For any new datum, the estimateof a∗ is updated dynamically.
• Cases of time-varying a∗ are“naturally” handled.
• Low complexity is required forstreaming applications.
• Fast convergence / Tracking.
• Large potential in Big Dataapplications
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 13/58
![Page 38: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/38.jpg)
Sparsity-Promoting Methods
`0-norm constrained minimization
• `0 (pseudo) norm minimization: NP-hard nonconvex task.
• a : mina∈Rl ‖a‖0, s.t. ‖y − Ua‖22 ≤ ε
• The above is carried out via greedy-type algorithmic arguments.
Constrained Least Squares Estimation: Three equivalent formulations
• a := argmina∈Rl
{‖y − Ua‖22 + λ‖a‖1
}• a : mina∈Rl ‖y − Ua‖22, s.t. ‖a‖1 ≤ ρ
• a : mina∈Rl ‖a‖1, s.t. ‖y − Ua‖22 ≤ ε• Why `1 norm: It is the “closest” to `0 “norm” (number of
nonzero elements) that retains its convex nature.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 14/58
![Page 39: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/39.jpg)
Sparsity-Promoting Methods
`0-norm constrained minimization
• `0 (pseudo) norm minimization: NP-hard nonconvex task.
• a : mina∈Rl ‖a‖0, s.t. ‖y − Ua‖22 ≤ ε
• The above is carried out via greedy-type algorithmic arguments.
Constrained Least Squares Estimation: Three equivalent formulations
• a := argmina∈Rl
{‖y − Ua‖22 + λ‖a‖1
}• a : mina∈Rl ‖y − Ua‖22, s.t. ‖a‖1 ≤ ρ
• a : mina∈Rl ‖a‖1, s.t. ‖y − Ua‖22 ≤ ε• Why `1 norm: It is the “closest” to `0 “norm” (number of
nonzero elements) that retains its convex nature.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 14/58
![Page 40: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/40.jpg)
Sparsity-Promoting Methods
`0-norm constrained minimization
• `0 (pseudo) norm minimization: NP-hard nonconvex task.
• a : mina∈Rl ‖a‖0, s.t. ‖y − Ua‖22 ≤ ε
• The above is carried out via greedy-type algorithmic arguments.
Constrained Least Squares Estimation: Three equivalent formulations
• a := argmina∈Rl
{‖y − Ua‖22 + λ‖a‖1
}• a : mina∈Rl ‖y − Ua‖22, s.t. ‖a‖1 ≤ ρ
• a : mina∈Rl ‖a‖1, s.t. ‖y − Ua‖22 ≤ ε• Why `1 norm: It is the “closest” to `0 “norm” (number of
nonzero elements) that retains its convex nature.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 14/58
![Page 41: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/41.jpg)
Sparsity-Promoting Methods
`0-norm constrained minimization
• `0 (pseudo) norm minimization: NP-hard nonconvex task.
• a : mina∈Rl ‖a‖0, s.t. ‖y − Ua‖22 ≤ ε
• The above is carried out via greedy-type algorithmic arguments.
Constrained Least Squares Estimation: Three equivalent formulations
• a := argmina∈Rl
{‖y − Ua‖22 + λ‖a‖1
}• a : mina∈Rl ‖y − Ua‖22, s.t. ‖a‖1 ≤ ρ
• a : mina∈Rl ‖a‖1, s.t. ‖y − Ua‖22 ≤ ε• Why `1 norm: It is the “closest” to `0 “norm” (number of
nonzero elements) that retains its convex nature.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 14/58
![Page 42: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/42.jpg)
Sparsity-Promoting Methods
`0-norm constrained minimization
• `0 (pseudo) norm minimization: NP-hard nonconvex task.
• a : mina∈Rl ‖a‖0, s.t. ‖y − Ua‖22 ≤ ε
• The above is carried out via greedy-type algorithmic arguments.
Constrained Least Squares Estimation: Three equivalent formulations
• a := argmina∈Rl
{‖y − Ua‖22 + λ‖a‖1
}• a : mina∈Rl ‖y − Ua‖22, s.t. ‖a‖1 ≤ ρ
• a : mina∈Rl ‖a‖1, s.t. ‖y − Ua‖22 ≤ ε• Why `1 norm: It is the “closest” to `0 “norm” (number of
nonzero elements) that retains its convex nature.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 14/58
![Page 43: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/43.jpg)
Sparsity-Promoting Methods
Hard and Soft thresholding
• The `1 norm is associated with a soft thresholding operation on therespective coefficients. This is a continuous function operation, butit adds bias even for the large values. On the other hand, hardthresholding is a discontinuous one.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 15/58
![Page 44: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/44.jpg)
Batch Penalized Least-Squares Estimator
Penalized Least-Squares - General Case
mina∈RL
{1
2‖y −Ua‖22 + λ
L∑i=1
p(|ai|)
}• p(·), sparsity-promoting penalty function,
• λ, regularization parameter.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 16/58
![Page 45: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/45.jpg)
Batch Penalized Least-Squares Estimator
Penalized Least-Squares - General Case
mina∈RL
{1
2‖y −Ua‖22 + λ
L∑i=1
p(|ai|)
}• p(·), sparsity-promoting penalty function,
• λ, regularization parameter.
Examples: Penalty functions
• p(|ai|) := |ai|γ , ∀ai ∈ R
• p(|ai|) = λ(1− e−β|ai|
)• p(|ai|) := λ
log(γ+1) log(γ|ai|+ 1), ∀ai ∈ R
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 16/58
![Page 46: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/46.jpg)
Online Sparsity-Promoting Methods
Penalized Recursive LS
mina∈RL
{1
2
N∑n=1
βN−ne2n + λL∑i=1
p(|ai|)
},
rn+1 = βrn + yn+1un+1, Rn+1 = βRn + un+1uTn+1
an+1 = f(rn+1,Rn+1)
• It Works!
• Complexity O(L2)
• Regularization parameter needs fine tuning
• [Angelosante, Bazerque and Giannakis, 2010]
• [Eksioglu and Tanc, 2011]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 17/58
![Page 47: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/47.jpg)
Online Sparsity-Promoting Methods
Penalized Recursive LS
mina∈RL
{1
2
N∑n=1
βN−ne2n + λL∑i=1
p(|ai|)
},
rN :=
N∑n=1
βN−nynun, RN :=
N∑n=1
βN−nunuTn
rn+1 = βrn + yn+1un+1, Rn+1 = βRn + un+1uTn+1
an+1 = f(rn+1,Rn+1)
• It Works!
• Complexity O(L2)
• Regularization parameter needs fine tuning
• [Angelosante, Bazerque and Giannakis, 2010]
• [Eksioglu and Tanc, 2011]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 17/58
![Page 48: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/48.jpg)
Online Sparsity-Promoting Methods
Penalized Recursive LS
mina∈RL
{1
2
N∑n=1
βN−ne2n + λL∑i=1
p(|ai|)
},
rn+1 = βrn + yn+1un+1, Rn+1 = βRn + un+1uTn+1
an+1 = f(rn+1,Rn+1)
• It Works!
• Complexity O(L2)
• Regularization parameter needs fine tuning
• [Angelosante, Bazerque and Giannakis, 2010]
• [Eksioglu and Tanc, 2011]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 17/58
![Page 49: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/49.jpg)
Online Sparsity-Promoting Methods
Penalized Recursive LS
mina∈RL
{1
2
N∑n=1
βN−ne2n + λL∑i=1
p(|ai|)
},
rn+1 = βrn + yn+1un+1, Rn+1 = βRn + un+1uTn+1
an+1 = f(rn+1,Rn+1)
• It Works!
• Complexity O(L2)
• Regularization parameter needs fine tuning
• [Angelosante, Bazerque and Giannakis, 2010]
• [Eksioglu and Tanc, 2011]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 17/58
![Page 50: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/50.jpg)
Online Sparsity-Promoting Methods
Penalized Recursive LS
mina∈RL
{1
2
N∑n=1
βN−ne2n + λL∑i=1
p(|ai|)
},
rn+1 = βrn + yn+1un+1, Rn+1 = βRn + un+1uTn+1
an+1 = f(rn+1,Rn+1)
• It Works!
• Complexity O(L2)
• Regularization parameter needs fine tuning
• [Angelosante, Bazerque and Giannakis, 2010]
• [Eksioglu and Tanc, 2011]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 17/58
![Page 51: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/51.jpg)
Online Sparsity-Promoting Methods
Penalized stochastic gradient descent: LMS type
mina∈RL
{1
2e2n + λ
L∑i=1
p(|ai|)
}
an+1 = an + µen(a)un − µλf(an)
f(an) =
[∂p(|an,1|)∂an,1
,∂p(|an,2|)∂an,2
, . . . ,∂p(|an,L|)∂an,L
]T• Complexity O(L)• It Works! (when it is compared to standard LMS)• Slow convergence• Regularization parameter needs fine tuning
• [Chen, Gu and Hero, 2009]• [Mileounis, Babadi, Kalouptsidis and Tarokh, 2010]• [Wang and Gu, 2012]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 18/58
![Page 52: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/52.jpg)
Online Sparsity-Promoting Methods
Penalized stochastic gradient descent: LMS type
mina∈RL
{1
2e2n + λ
L∑i=1
p(|ai|)
}
an+1 = an + µen(a)un − µλf(an)
f(an) =
[∂p(|an,1|)∂an,1
,∂p(|an,2|)∂an,2
, . . . ,∂p(|an,L|)∂an,L
]T• Complexity O(L)• It Works! (when it is compared to standard LMS)• Slow convergence• Regularization parameter needs fine tuning
• [Chen, Gu and Hero, 2009]• [Mileounis, Babadi, Kalouptsidis and Tarokh, 2010]• [Wang and Gu, 2012]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 18/58
![Page 53: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/53.jpg)
Online Sparsity-Promoting Methods
Penalized stochastic gradient descent: LMS type
mina∈RL
{1
2e2n + λ
L∑i=1
p(|ai|)
}
an+1 = an + µen(a)un − µλf(an)
f(an) =
[∂p(|an,1|)∂an,1
,∂p(|an,2|)∂an,2
, . . . ,∂p(|an,L|)∂an,L
]T• Complexity O(L)• It Works! (when it is compared to standard LMS)• Slow convergence• Regularization parameter needs fine tuning
• [Chen, Gu and Hero, 2009]• [Mileounis, Babadi, Kalouptsidis and Tarokh, 2010]• [Wang and Gu, 2012]
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 18/58
![Page 54: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/54.jpg)
The Set-Theoretic Estimation Approach
The main concept
A descendent of POCS
Projection onto a Closed Convex Set
Let C be a closed convex set in RL. Then, for each a ∈ RL there exists aunique a∗ ∈ C such that
‖a− a∗‖ = ming∈C‖a− g‖.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 19/58
![Page 55: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/55.jpg)
The Set-Theoretic Estimation Approach
The main concept
A descendent of POCS
Projection onto a Closed Convex Set
Let C be a closed convex set in RL. Then, for each a ∈ RL there exists aunique a∗ ∈ C such that
‖a− a∗‖ = ming∈C‖a− g‖.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 19/58
![Page 56: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/56.jpg)
The Set-Theoretic Estimation Approach
The main concept
A descendent of POCS
Projection onto a Closed Convex Set
Let C be a closed convex set in RL. Then, for each a ∈ RL there exists aunique a∗ ∈ C such that
‖a− a∗‖ = ming∈C‖a− g‖.
Metric Projection Mapping
Metric Projection is the mappingPC : RL → C : a 7→ PC(a) := a∗.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 19/58
![Page 57: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/57.jpg)
The Set-Theoretic Estimation Approach
The main concept
A descendent of POCS
Projection onto a Closed Convex Set
Let C be a closed convex set in RL. Then, for each a ∈ RL there exists aunique a∗ ∈ C such that
‖a− a∗‖ = ming∈C‖a− g‖.
Metric Projection Mapping
Metric Projection is the mappingPC : RL → C : a 7→ PC(a) := a∗.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 19/58
![Page 58: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/58.jpg)
The Set-Theoretic Estimation Approach
The main concept
A descendent of POCS
Projection onto a Closed Convex Set
Let C be a closed convex set in RL. Then, for each a ∈ RL there exists aunique a∗ ∈ C such that
‖a− a∗‖ = ming∈C‖a− g‖.
Metric Projection Mapping
Metric Projection is the mappingPC : RL → C : a 7→ PC(a) := a∗.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 19/58
![Page 59: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/59.jpg)
The Set-Theoretic Estimation Approach
The main concept
A descendent of POCS
Projection onto a Closed Convex Set
Let C be a closed convex set in RL. Then, for each a ∈ RL there exists aunique a∗ ∈ C such that
‖a− a∗‖ = ming∈C‖a− g‖.
Relaxed Projection Mapping
The relaxed Projection is the mappingTC(a) := a+ µ(PC(a)− a),µ ∈ (0, 2),∀a ∈ RL.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 19/58
![Page 60: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/60.jpg)
The Set-Theoretic Estimation Approach
The POCS: Finite number of Convex Sets [Von Neumann ’33], [Bregman’65], [Gubin, Polyak, Raik ’67]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂qi=1 Ci 6= ∅, let
their associated projection mappings be PC1, . . . , PCq
. For any a ∈ RL, definethe sequence of projections:
PC1(a).
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 20/58
![Page 61: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/61.jpg)
The Set-Theoretic Estimation Approach
The POCS: Finite number of Convex Sets [Von Neumann ’33], [Bregman’65], [Gubin, Polyak, Raik ’67]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂qi=1 Ci 6= ∅, let
their associated projection mappings be PC1, . . . , PCq
. For any a ∈ RL, definethe sequence of projections:
PC1(a).
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 20/58
![Page 62: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/62.jpg)
The Set-Theoretic Estimation Approach
The POCS: Finite number of Convex Sets [Von Neumann ’33], [Bregman’65], [Gubin, Polyak, Raik ’67]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂qi=1 Ci 6= ∅, let
their associated projection mappings be PC1, . . . , PCq
. For any a ∈ RL, definethe sequence of projections:
PC2PC1
(a).
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 20/58
![Page 63: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/63.jpg)
The Set-Theoretic Estimation Approach
The POCS: Finite number of Convex Sets [Von Neumann ’33], [Bregman’65], [Gubin, Polyak, Raik ’67]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂qi=1 Ci 6= ∅, let
their associated projection mappings be PC1, . . . , PCq
. For any a ∈ RL, definethe sequence of projections:
PC1PC2
PC1(a).
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 20/58
![Page 64: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/64.jpg)
The Set-Theoretic Estimation Approach
The POCS: Finite number of Convex Sets [Von Neumann ’33], [Bregman’65], [Gubin, Polyak, Raik ’67]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂qi=1 Ci 6= ∅, let
their associated projection mappings be PC1, . . . , PCq
. For any a ∈ RL, definethe sequence of projections:
PC2PC1
PC2PC1
(a).
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 20/58
![Page 65: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/65.jpg)
The Set-Theoretic Estimation Approach
The POCS: Finite number of Convex Sets [Von Neumann ’33], [Bregman’65], [Gubin, Polyak, Raik ’67]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂qi=1 Ci 6= ∅, let
their associated projection mappings be PC1, . . . , PCq
. For any a ∈ RL, definethe sequence of projections:
· · ·PC2PC1
PC2PC1
(a).
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 20/58
![Page 66: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/66.jpg)
The Set-Theoretic Estimation Approach
Convex Combination of Projection Mappings [Pierra ’84]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂q
i=1 Ci 6= ∅, let theirassociated projection mappings be PC1 , . . . , PCq . Let also a set of positive constantsw1, . . . , wq such that
∑qi=1 wi = 1. Then for any a0, the sequence
an+1 = an + µn(
q∑i=1
wiPCi(an)︸ ︷︷ ︸Convex combination of projections
−an), ∀n,
converges weakly to a point a∗ in⋂q
i=1 Ci,where µn ∈ (ε,Mn), for ε ∈ (0, 1), and
Mn :=∑q
i=1 wi‖PCi(an)−an‖2
‖∑q
i=1 wiPCi(an)−an‖2
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 21/58
![Page 67: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/67.jpg)
The Set-Theoretic Estimation Approach
Convex Combination of Projection Mappings [Pierra ’84]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂q
i=1 Ci 6= ∅, let theirassociated projection mappings be PC1 , . . . , PCq . Let also a set of positive constantsw1, . . . , wq such that
∑qi=1 wi = 1. Then for any a0, the sequence
an+1 = an + µn(
q∑i=1
wiPCi(an)︸ ︷︷ ︸Convex combination of projections
−an), ∀n,
converges weakly to a point a∗ in⋂q
i=1 Ci,where µn ∈ (ε,Mn), for ε ∈ (0, 1), and
Mn :=∑q
i=1 wi‖PCi(an)−an‖2
‖∑q
i=1 wiPCi(an)−an‖2
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 21/58
![Page 68: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/68.jpg)
The Set-Theoretic Estimation Approach
Convex Combination of Projection Mappings [Pierra ’84]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂q
i=1 Ci 6= ∅, let theirassociated projection mappings be PC1 , . . . , PCq . Let also a set of positive constantsw1, . . . , wq such that
∑qi=1 wi = 1. Then for any a0, the sequence
an+1 = an + µn(
q∑i=1
wiPCi(an)︸ ︷︷ ︸Convex combination of projections
−an), ∀n,
converges weakly to a point a∗ in⋂q
i=1 Ci,where µn ∈ (ε,Mn), for ε ∈ (0, 1), and
Mn :=∑q
i=1 wi‖PCi(an)−an‖2
‖∑q
i=1 wiPCi(an)−an‖2
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 21/58
![Page 69: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/69.jpg)
The Set-Theoretic Estimation Approach
Convex Combination of Projection Mappings [Pierra ’84]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂q
i=1 Ci 6= ∅, let theirassociated projection mappings be PC1 , . . . , PCq . Let also a set of positive constantsw1, . . . , wq such that
∑qi=1 wi = 1. Then for any a0, the sequence
an+1 = an + µn(
q∑i=1
wiPCi(an)︸ ︷︷ ︸Convex combination of projections
−an), ∀n,
converges weakly to a point a∗ in⋂q
i=1 Ci,where µn ∈ (ε,Mn), for ε ∈ (0, 1), and
Mn :=∑q
i=1 wi‖PCi(an)−an‖2
‖∑q
i=1 wiPCi(an)−an‖2
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 21/58
![Page 70: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/70.jpg)
The Set-Theoretic Estimation Approach
Convex Combination of Projection Mappings [Pierra ’84]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂q
i=1 Ci 6= ∅, let theirassociated projection mappings be PC1 , . . . , PCq . Let also a set of positive constantsw1, . . . , wq such that
∑qi=1 wi = 1. Then for any a0, the sequence
an+1 = an + µn(
q∑i=1
wiPCi(an)︸ ︷︷ ︸Convex combination of projections
−an), ∀n,
converges weakly to a point a∗ in⋂q
i=1 Ci,where µn ∈ (ε,Mn), for ε ∈ (0, 1), and
Mn :=∑q
i=1 wi‖PCi(an)−an‖2
‖∑q
i=1 wiPCi(an)−an‖2
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 21/58
![Page 71: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/71.jpg)
The Set-Theoretic Estimation Approach
Convex Combination of Projection Mappings [Pierra ’84]
Given a finite number of closed convex sets C1, . . . , Cq, with⋂q
i=1 Ci 6= ∅, let theirassociated projection mappings be PC1 , . . . , PCq . Let also a set of positive constantsw1, . . . , wq such that
∑qi=1 wi = 1. Then for any a0, the sequence
an+1 = an + µn(
q∑i=1
wiPCi(an)︸ ︷︷ ︸Convex combination of projections
−an), ∀n,
converges weakly to a point a∗ in⋂q
i=1 Ci,where µn ∈ (ε,Mn), for ε ∈ (0, 1), and
Mn :=∑q
i=1 wi‖PCi(an)−an‖2
‖∑q
i=1 wiPCi(an)−an‖2
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 21/58
![Page 72: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/72.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 73: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/73.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Find a point in the intersection of all the hyperslabs
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 74: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/74.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Find a point in the intersection of all the hyperslabs
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 75: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/75.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Find a point in the intersection of all the hyperslabs
1
1
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 76: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/76.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Find a point in the intersection of all the hyperslabs
2
2
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 77: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/77.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Find a point in the intersection of all the hyperslabs
3
3
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 78: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/78.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Find a point in the intersection of all the hyperslabs4
4
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 79: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/79.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
Find a point in the intersection of all the hyperslabs5
5
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 80: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/80.jpg)
Set-Theoretic Estimation: The Online Case Approach
Constructing the Convex Sets
For each received set of measurements (training pairs) (un, yn), construct ahyperslab:
Sn[ε] :={a ∈ RL : |uTna− yn| ≤ ε
}
Solution
[Yamada 2001], [Yamada, Slavakis, Yamada 2002], [Yamada, Ogura 2004],[Slavakis, Yamada Ogura 2006].
[Chouvardas, Slavakis, Theodoridis, Yamada, 2013]: Under the assumption ofBounded noise it converges with probability 1 arbitrarily close to the truemodel.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 22/58
![Page 81: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/81.jpg)
Adaptive Projection Subgradient Method (APSM)
The Algorithm
an+1 := an + µn
n∑i=n−q+1
ω(n)i
(PSn[ε](an)− an
)Projection onto Hyperslab
PSn[ε](a) = a+
yn−ε−uT
na‖un‖2 un, if yn − ε > uTna
0, if |uTna− yn| ≤ εyn+ε−uT
na‖un‖2 un, if yn + ε < uTna
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 23/58
![Page 82: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/82.jpg)
Adaptive Projection Subgradient Method (APSM)
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 24/58
![Page 83: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/83.jpg)
Adaptive Projection Subgradient Method (APSM)
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 24/58
![Page 84: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/84.jpg)
Adaptive Projection Subgradient Method (APSM)
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 24/58
![Page 85: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/85.jpg)
Adaptive Projection Subgradient Method (APSM)
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 24/58
![Page 86: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/86.jpg)
APSM under the `1 ball constraint
The `1-ball case
Given (un, yn), n = 0, 1, 2, . . ., find a such that∣∣aTun − yn∣∣ ≤ ε, n = 0, 1, 2, . . .
‖a‖1 ≤ δ.
The recursion:
an+1 := PB`1[δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
,
converges to
a∗ ∈ B`1 [δ] ∩
⋂n≥n0
Sn[ε]
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 25/58
![Page 87: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/87.jpg)
APSM under the `1 ball constraint
The `1-ball case
Given (un, yn), n = 0, 1, 2, . . ., find a such that∣∣aTun − yn∣∣ ≤ ε, n = 0, 1, 2, . . .
‖a‖1 ≤ δ.
The recursion:
an+1 := PB`1[δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
,
converges to
a∗ ∈ B`1 [δ] ∩
⋂n≥n0
Sn[ε]
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 25/58
![Page 88: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/88.jpg)
APSM under the `1 ball constraint
The `1-ball case
Given (un, yn), n = 0, 1, 2, . . ., find a such that∣∣aTun − yn∣∣ ≤ ε, n = 0, 1, 2, . . .
‖a‖1 ≤ δ.
The recursion:
an+1 := PB`1[δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
,
converges to
a∗ ∈ B`1 [δ] ∩
⋂n≥n0
Sn[ε]
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 25/58
![Page 89: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/89.jpg)
APSM under the `1 ball constraint
The `1-ball case
Given (un, yn), n = 0, 1, 2, . . ., find a such that∣∣aTun − yn∣∣ ≤ ε, n = 0, 1, 2, . . .
‖a‖1 ≤ δ.
The recursion:
an+1 := PB`1[δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
,
converges to
a∗ ∈ B`1 [δ] ∩
⋂n≥n0
Sn[ε]
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 25/58
![Page 90: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/90.jpg)
APSM under the `1 ball constraint
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 26/58
![Page 91: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/91.jpg)
APSM under the `1 ball constraint
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 26/58
![Page 92: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/92.jpg)
APSM under the `1 ball constraint
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 26/58
![Page 93: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/93.jpg)
APSM under the `1 ball constraint
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 26/58
![Page 94: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/94.jpg)
APSM under the `1 ball constraint
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 26/58
![Page 95: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/95.jpg)
APSM under the `1 ball constraint
Geometric illustration example
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 26/58
![Page 96: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/96.jpg)
APSM under the weighted `1 ball constraint
The weighted `1-ball case:
• Convergence can be significantly speeded up if `1-ball, is replacedby the weighted `1 ball.
• Definition:
‖a‖1,w :=
L∑i=1
wi|ai|.
• Time-adaptive weighted norm:
wn,i :=1
|an,i|+ ε′n.
• A time varying constraint case.
• The recursion:
an+1 := PB`1[wn,δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 27/58
![Page 97: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/97.jpg)
APSM under the weighted `1 ball constraint
The weighted `1-ball case:
• Convergence can be significantly speeded up if `1-ball, is replacedby the weighted `1 ball.
• Definition:
‖a‖1,w :=
L∑i=1
wi|ai|.
• Time-adaptive weighted norm:
wn,i :=1
|an,i|+ ε′n.
• A time varying constraint case.
• The recursion:
an+1 := PB`1[wn,δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 27/58
![Page 98: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/98.jpg)
APSM under the weighted `1 ball constraint
The weighted `1-ball case:
• Convergence can be significantly speeded up if `1-ball, is replacedby the weighted `1 ball.
• Definition:
‖a‖1,w :=
L∑i=1
wi|ai|.
• Time-adaptive weighted norm:
wn,i :=1
|an,i|+ ε′n.
• A time varying constraint case.
• The recursion:
an+1 := PB`1[wn,δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 27/58
![Page 99: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/99.jpg)
APSM under the weighted `1 ball constraint
The weighted `1-ball case:
• Convergence can be significantly speeded up if `1-ball, is replacedby the weighted `1 ball.
• Definition:
‖a‖1,w :=
L∑i=1
wi|ai|.
• Time-adaptive weighted norm:
wn,i :=1
|an,i|+ ε′n.
• A time varying constraint case.
• The recursion:
an+1 := PB`1[wn,δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 27/58
![Page 100: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/100.jpg)
APSM under the weighted `1 ball constraint
The weighted `1-ball case:
• Convergence can be significantly speeded up if `1-ball, is replacedby the weighted `1 ball.
• Definition:
‖a‖1,w :=
L∑i=1
wi|ai|.
• Time-adaptive weighted norm:
wn,i :=1
|an,i|+ ε′n.
• A time varying constraint case.
• The recursion:
an+1 := PB`1[wn,δ]
an + µn
n∑j=n−q+1
ω(n)j PSj [ε](an)− an
.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 27/58
![Page 101: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/101.jpg)
APSM under the weighted `1 ball constraint
Geometric illustration example
_
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 28/58
![Page 102: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/102.jpg)
APSM under the weighted `1 ball constraint
Geometric illustration example
_
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 28/58
![Page 103: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/103.jpg)
APSM under the weighted `1 ball constraint
Geometric illustration example
_
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 28/58
![Page 104: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/104.jpg)
APSM under the weighted `1 ball constraint
Geometric illustration example
_
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 28/58
![Page 105: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/105.jpg)
APSM under the weighted `1 ball constraint
Convergence of the Scheme
• Does this scheme converge?Note that our constraint, i.e., the weighted `1-ball is atime-varying constraint.Remark: This case was not covered by the existing theory.
0
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 29/58
![Page 106: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/106.jpg)
APSM under the weighted `1 ball constraint
Convergence of the Scheme
• Does this scheme converge?Note that our constraint, i.e., the weighted `1-ball is atime-varying constraint.Remark: This case was not covered by the existing theory.
0
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 29/58
![Page 107: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/107.jpg)
APSM under the weighted `1 ball constraint
Convergence of the Scheme
• Does this scheme converge?Note that our constraint, i.e., the weighted `1-ball is atime-varying constraint.Remark: This case was not covered by the existing theory.
0
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 29/58
![Page 108: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/108.jpg)
APSM under the weighted `1 ball constraint
Convergence of the Scheme
• Does this scheme converge?Note that our constraint, i.e., the weighted `1-ball is atime-varying constraint.Remark: This case was not covered by the existing theory.
0
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 29/58
![Page 109: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/109.jpg)
APSM under the weighted `1 ball constraint
Convergence of the Scheme
• Does this scheme converge?Note that our constraint, i.e., the weighted `1-ball is atime-varying constraint.Remark: This case was not covered by the existing theory.
0
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 29/58
![Page 110: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/110.jpg)
APSM under the weighted `1 ball constraint
Convergence of the Scheme
• Does this scheme converge?Note that our constraint, i.e., the weighted `1-ball is atime-varying constraint.Remark: This case was not covered by the existing theory.
0
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 29/58
![Page 111: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/111.jpg)
Simulation Examples
Example: Time-invariant signal sparse in wavelet domain
L := 1024, ‖a∗‖0 := 100 wavelet coefficients. The radius of the `1-ball is set to δ := 101.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 30/58
![Page 112: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/112.jpg)
Simulation Examples
Example: Time varying signal compressible in wavelet domain
L := 4096.The sum of two chirp signals.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 31/58
![Page 113: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/113.jpg)
Simulation Examples
Example: Time varying signal compressible in wavelet domain
L := 4096. The radius of the `1-ball is set to δ := 40.
Movies of the OCCD, and the APWL1sub.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 32/58
![Page 114: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/114.jpg)
Generalized Thresholding Rules
Thresholding rules associated with non-convex penalty functions
• Penalized LS thresholding operators:
mina
1
2(a− a)2 + λp(|a|)
• p(·): nonnegative, nondecreasing and differentiable function on(0,∞)
• Under some general conditions it has a unique solution [Antoniadis2007].
• PLSTO basically defines a mapping
a 7→ mina
1
2(a− a)2 + λp(|a|)
which corresponds to a Shrinkage operator.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 33/58
![Page 115: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/115.jpg)
Generalized Thresholding Rules
Thresholding rules associated with non-convex penalty functions
• Penalized LS thresholding operators:
mina
1
2(a− a)2 + λp(|a|)
• p(·): nonnegative, nondecreasing and differentiable function on(0,∞)
• Under some general conditions it has a unique solution [Antoniadis2007].
• PLSTO basically defines a mapping
a 7→ mina
1
2(a− a)2 + λp(|a|)
which corresponds to a Shrinkage operator.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 33/58
![Page 116: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/116.jpg)
Generalized Thresholding Rules
Thresholding rules associated with non-convex penalty functions
• Penalized LS thresholding operators:
mina
1
2(a− a)2 + λp(|a|)
• p(·): nonnegative, nondecreasing and differentiable function on(0,∞)
• Under some general conditions it has a unique solution [Antoniadis2007].
• PLSTO basically defines a mapping
a 7→ mina
1
2(a− a)2 + λp(|a|)
which corresponds to a Shrinkage operator.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 33/58
![Page 117: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/117.jpg)
Generalized Thresholding Rules
Thresholding rules associated with non-convex penalty functions
• Penalized LS thresholding operators:
mina
1
2(a− a)2 + λp(|a|)
• p(·): nonnegative, nondecreasing and differentiable function on(0,∞)
• Under some general conditions it has a unique solution [Antoniadis2007].
• PLSTO basically defines a mapping
a 7→ mina
1
2(a− a)2 + λp(|a|)
which corresponds to a Shrinkage operator.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 33/58
![Page 118: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/118.jpg)
Generalized Thresholding Rules
Examples: Penalty functions
• p(|a|) := |a|γ , ∀a ∈ R
• p(|a|) = λ(1− e−β|a|
)• p(|a|) := λ
log(γ+1) log(γ|a|+ 1), ∀a ∈ R
Examples: Penalized Least-Squares Thresholding Operators
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 34/58
![Page 119: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/119.jpg)
Generalized Thresholding Rules
Examples: Penalty functions
• p(|a|) := |a|γ , ∀a ∈ R
• p(|a|) = λ(1− e−β|a|
)• p(|a|) := λ
log(γ+1) log(γ|a|+ 1), ∀a ∈ R
Examples: Penalized Least-Squares Thresholding Operators
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 34/58
![Page 120: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/120.jpg)
Generalized Thresholding Rules
Generalized Thresholding (GT) operator: Definition:
For any a ∈ RL, z := T(K)GT (a) is obtained coordinate-wise:
∀l ∈ 1, L, zl :=
{al, If, al is one of the largest K components,
shr(al), otherwise
Shrinkage Function (Shr)
• τshr(τ) ≥ 0, ∀τ ∈ R.
• shr acts as a strict shrinkage operator over all intervals which do notinclude 0.
• Any arbitrary function inline with the properties above can be used.
• All the penalized Least-Squares thresholding operators are included.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 35/58
![Page 121: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/121.jpg)
Generalized Thresholding Rules
Generalized Thresholding (GT) operator: Definition:
For any a ∈ RL, z := T(K)GT (a) is obtained coordinate-wise:
∀l ∈ 1, L, zl :=
{al, If, al is one of the largest K components,
shr(al), otherwise
Shrinkage Function (Shr)
• τshr(τ) ≥ 0, ∀τ ∈ R.
• shr acts as a strict shrinkage operator over all intervals which do notinclude 0.
• Any arbitrary function inline with the properties above can be used.
• All the penalized Least-Squares thresholding operators are included.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 35/58
![Page 122: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/122.jpg)
Generalized Thresholding Rules
Generalized Thresholding (GT) operator: Definition:
For any a ∈ RL, z := T(K)GT (a) is obtained coordinate-wise:
∀l ∈ 1, L, zl :=
{al, If, al is one of the largest K components,
shr(al), otherwise
In words
• Choose the largest K components of the estimate.
• The rest are shrunk according to the shrinkage rule.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 35/58
![Page 123: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/123.jpg)
Generalized Thresholding Rules
Generalized Thresholding (GT) operator: Definition:
For any a ∈ RL, z := T(K)GT (a) is obtained coordinate-wise:
∀l ∈ 1, L, zl :=
{al, If, al is one of the largest K components,
shr(al), otherwise
Examples: Generalized Thresholding (GT) operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 35/58
![Page 124: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/124.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 125: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/125.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 126: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/126.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 127: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/127.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*P ( )
P ( )
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 128: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/128.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*P ( )
P ( )
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 129: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/129.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
**
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 130: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/130.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
***
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 131: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/131.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
=0
**
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 132: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/132.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
=0shr( )
**
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 133: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/133.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 134: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/134.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 135: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/135.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 136: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/136.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 137: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/137.jpg)
Adaptive Projection-Based Algorithm With GeneralizedThresholding (APGT)
The Algorithm
an+1 := Tn
an + µn
n∑i=n−q+1
ω(n)i (P (an)− an)
• Each piece of a-priori information, is also represented by a set
*
=0
=0
Thresholding Operator
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 36/58
![Page 138: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/138.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.∀x ∈ RL,∃Yx ⊂ Fix(T ) : ∀y ∈ Yx,‖T (x)− y‖ ≤ ‖x− y‖
• The fixed point set of GT is a union ofsubspaces (non-convex).
Examples: Union of Subspaces for s = 2
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 37/58
![Page 139: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/139.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.∀x ∈ RL,∃Yx ⊂ Fix(T ) : ∀y ∈ Yx,‖T (x)− y‖ ≤ ‖x− y‖
• The fixed point set of GT is a union ofsubspaces (non-convex).
Examples: Union of Subspaces for s = 2
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 37/58
![Page 140: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/140.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.∀x ∈ RL,∃Yx ⊂ Fix(T ) : ∀y ∈ Yx,‖T (x)− y‖ ≤ ‖x− y‖
• The fixed point set of GT is a union ofsubspaces (non-convex).
Examples: Union of Subspaces for s = 2
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 37/58
![Page 141: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/141.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.∀x ∈ RL,∃Yx ⊂ Fix(T ) : ∀y ∈ Yx,‖T (x)− y‖ ≤ ‖x− y‖
• The fixed point set of GT is a union ofsubspaces (non-convex).
Examples: Union of Subspaces for s = 2
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 37/58
![Page 142: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/142.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.∀x ∈ RL,∃Yx ⊂ Fix(T ) : ∀y ∈ Yx,‖T (x)− y‖ ≤ ‖x− y‖
• The fixed point set of GT is a union ofsubspaces (non-convex).
Examples: Union of Subspaces for s = 2
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 37/58
![Page 143: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/143.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.∀x ∈ RL,∃Yx ⊂ Fix(T ) : ∀y ∈ Yx,‖T (x)− y‖ ≤ ‖x− y‖
• The fixed point set of GT is a union ofsubspaces (non-convex).
Examples: Union of Subspaces for s = 2
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 37/58
![Page 144: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/144.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.
• The fixed point set of GT is a union ofsubspaces (non-convex).
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 38/58
![Page 145: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/145.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.
• The fixed point set of GT is a union ofsubspaces (non-convex).
• It has been shown [Slavakis, Kopsinis, Theodoridis, McLaughlin, 2013]:
The algorithm leads to a sequence of estimates (a)n∈Z≥0whose set
of cluster points is nonempty,each one of the cluster points is guaranteed to be, at most, s-sparse,the solution is located arbitrarily close to an intersection of aninfinite number of hyperslabs.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 38/58
![Page 146: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/146.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.
• The fixed point set of GT is a union ofsubspaces (non-convex).
• It has been shown [Slavakis, Kopsinis, Theodoridis, McLaughlin, 2013]:
The algorithm leads to a sequence of estimates (a)n∈Z≥0whose set
of cluster points is nonempty,
each one of the cluster points is guaranteed to be, at most, s-sparse,the solution is located arbitrarily close to an intersection of aninfinite number of hyperslabs.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 38/58
![Page 147: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/147.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.
• The fixed point set of GT is a union ofsubspaces (non-convex).
• It has been shown [Slavakis, Kopsinis, Theodoridis, McLaughlin, 2013]:
The algorithm leads to a sequence of estimates (a)n∈Z≥0whose set
of cluster points is nonempty,each one of the cluster points is guaranteed to be, at most, s-sparse,
the solution is located arbitrarily close to an intersection of aninfinite number of hyperslabs.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 38/58
![Page 148: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/148.jpg)
Adaptive Projection-Based Algorithm With GT (APGT)
Convergence of APGT
• Partially Quasi-nonexpansive Mapping.
• The fixed point set of GT is a union ofsubspaces (non-convex).
• It has been shown [Slavakis, Kopsinis, Theodoridis, McLaughlin, 2013]:
The algorithm leads to a sequence of estimates (a)n∈Z≥0whose set
of cluster points is nonempty,each one of the cluster points is guaranteed to be, at most, s-sparse,the solution is located arbitrarily close to an intersection of aninfinite number of hyperslabs.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 38/58
![Page 149: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/149.jpg)
Simulation Examples
Example: Time-varying case exhibiting an abrupt change
L := 1024, s = 100 (up to n = 1500, and s = 110 afterwards)
APGT:O(qL+ qK)OSCD: O(L2)IPAPA: O(q3)
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 39/58
![Page 150: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/150.jpg)
Simulation Examples
Example: Sparse system identification with colored input
L := 600, s = 60, AR input (cond ' 100) .
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 40/58
![Page 151: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/151.jpg)
Bibliography• I. Yamada and N. Ogura. Adaptive Projected Subgradient Method for asymptotic minimization of sequence of
nonnegative convex functions. Numerical Functional Analysis and Optimization, 25(7&8), 2004.• K. Slavakis, I. Yamada, and N. Ogura. The adaptive projected subgradient method over the fixed point set of
strongly attracting nonexpansive mappings. Numerical Functional Analysis and Optimization, 27 (7&8), Nov 2006.• S. Theodoridis, K. Slavakis, and I. Yamada, “Adaptive learning in a world of projections: a unifying framework for
linear and nonlinear classification and regression tasks,” ,” IEEE Trans. Signal Proc., vol. 28, Jan. 2011.• K. Slavakis and I. Yamada. The adaptive projected subgradient method constrained by families of quasi-
nonexpansive mappings and its application to online learning. SIAM Journal on Optimization, vol. 23, no. 1, 2013.• Y. Kopsinis, K. Slavakis, and S. Theodoridis, “Online sparse system identification and signal reconstruction using
projections onto weighted `1 balls,” IEEE Trans. Signal Proc., vol. 59, Mar. 2011.• S. Chouvardas, K. Slavakis, and S. Theodoridis. Adaptive robust distributed learning in diffusion sensor networks.
IEEE Transactions on Signal Processing, vol. 59, Oct. 2011.• S. Chouvardas, K. Slavakis, Y. Kopsinis, and S. Theodoridis. A sparsity promoting adaptive algorithm for
distributed learning. IEEE Transactions on Signal Processing, vol. 60, Oct. 2012.• K. Slavakis, Y. Kopsinis, S. Theodoridis, and S. McLaughlin. Generalized thresholding and online sparsity-aware
learning in a union of subspaces. Accepted for publication in the IEEE Transactions on Signal Processing, 2013• S. Chouvardas, K. Slavakis, S. Theodoridis, and I. Yamada. Stochastic analysis of hyperslab-based adaptive
projected subgradient method under bounded noise. IEEE Signal Processing Letters, vol. 20, 2013.• Y. Kopsinis, K. Slavakis, S. Theodoridis, “Thresholding-Based Online Algorithms of Complexity Comparable to
Sparse LMS methods,” Under preparation (A part submitted to ISCAS 2013),
• D. Angelosante, J. A. Bazerque, and G. B. Giannakis, ‘Online Adaptive Estimation of Sparse Signals: Where RLSMeets the l1-Norm’, IEEE Transactions on Signal Processing, vol. 58, Jul. 2010.
• E. M. Eksioglu and A. K. Tanc, ‘RLS Algorithm With Convex Regularization’, IEEE Signal Processing Letters, vol.18, Aug. 2011.
• Y. Chen, Y. Gu, and A. O. Hero, ‘Sparse LMS for system identification’, in IEEE International Conference onAcoustics, Speech and Signal Processing, ICASSP 2009.
• X. Wang and Y. Gu, ‘Proof of Convergence and Performance Analysis for Sparse Recovery via Zero-PointAttracting Projection’, IEEE Transactions on Signal Processing, vol. 60, Aug. 2012.
• G. Mileounis, B.Babadi, N. Kalouptsidis, and V. Tarokh, ”An Adaptive Greedy Algorithm with Application toNonlinear Communications”, IEEE Trans. on Signal Processing, vol. 58, Jun. 2010.
• P. Di Lorenzo and A. H. Sayed, “Sparse distributed learning based on diffusion adaptation,” IEEE Trans. SignalProcessing, vol. 61, March 2013.
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 41/58
![Page 152: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/152.jpg)
“Machine Learning: A Bayesian and Optimization Perspective”
by
Sergios Theodoridis
Academic Press, 2015
1050 pages
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 42/58
![Page 153: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/153.jpg)
Thank youfor your patience...
I hope that there are NO QUESTIONS !!!!!!!!!
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 43/58
![Page 154: An Online Learning View via a Projections' Path in the Sparse-landicore/Docs/Sergios... · 2016. 9. 28. · An Online Learning View via a Projections’ Path in the Sparse-land Sergios](https://reader035.fdocuments.in/reader035/viewer/2022071608/61471860f4263007b135996b/html5/thumbnails/154.jpg)
Thank youfor your patience...
I hope that there are NO QUESTIONS !!!!!!!!!
Sergios Theodoridis, University of Athens. An Online Learning View via a Projections’ Path in the Sparse-land, 43/58