Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint...
Transcript of Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint...
![Page 1: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/1.jpg)
Randomized iterative methods for linear systems
Robert Mansel Gower
IMA Leslie Fox Prize Meeting, Strathclyde, June 2017
![Page 2: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/2.jpg)
Motivation
![Page 3: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/3.jpg)
Large scale Kernel Ridge Regression
Problem: a9a
Origin: LIBSVM
![Page 4: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/4.jpg)
Large scale Kernel Ridge Regression
Problem: a9a
Origin: LIBSVM
Conjugate Gradient
![Page 5: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/5.jpg)
Large scale Kernel Ridge Regression
Problem: a9a
Origin: LIBSVM
Conjugate Gradient
Block Coordinate
Descent
![Page 6: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/6.jpg)
Large scale Kernel Ridge Regression
Problem: a9a
Origin: LIBSVM
Conjugate Gradient
Rademacher Sketch?
Block Coordinate
Descent
![Page 7: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/7.jpg)
Large scale Kernel Ridge Regression
Problem: a9a
Origin: LIBSVM
Conjugate Gradient
Rademacher Sketch?
Block Coordinate
Descent
Good enough
![Page 8: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/8.jpg)
Large scale Kernel Ridge Regression
Problem: a9a
Origin: LIBSVM
Conjugate Gradient
Cheikh S. Toure
Rademacher Sketch?
Block Coordinate
Descent
Good enough
![Page 9: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/9.jpg)
Linear Systems
![Page 10: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/10.jpg)
The Problem
Assumption: The system is consistent (i.e., has a solution)
![Page 11: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/11.jpg)
The Problem
![Page 12: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/12.jpg)
The Problem
B: Symmetric and positive definite
![Page 13: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/13.jpg)
The Problem
B: Symmetric and positive definite
As there are possibly multiple solutions, we compute the solution with the least B-norm.
![Page 14: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/14.jpg)
Randomized Methods
![Page 15: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/15.jpg)
Often fits in parallel/distributed architecture
Easy to analyse, good complexity
Easy to implement
Suitable for large scale problems: short recurrence, low iteration cost and low memory
The return of old methodsOld methods (Kaczmarz 1937, Gauss-Seidel
1823) make a randomized return, why?
Stochasticity inherent in problem
![Page 16: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/16.jpg)
Old Methods
![Page 17: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/17.jpg)
Randomized Kaczmarz T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
Karczmarz, M. S. (1937). Angenaherte Auflosung von Systemen linearer Gleichungen. Bulletin International de l’Académie Polonaise Des Sciences et Des Lettres, 35, 355–357.
![Page 18: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/18.jpg)
Randomized Kaczmarz T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
Karczmarz, M. S. (1937). Angenaherte Auflosung von Systemen linearer Gleichungen. Bulletin International de l’Académie Polonaise Des Sciences et Des Lettres, 35, 355–357.
![Page 19: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/19.jpg)
Randomized Kaczmarz T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
Karczmarz, M. S. (1937). Angenaherte Auflosung von Systemen linearer Gleichungen. Bulletin International de l’Académie Polonaise Des Sciences et Des Lettres, 35, 355–357.
![Page 20: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/20.jpg)
Randomized Kaczmarz T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
Karczmarz, M. S. (1937). Angenaherte Auflosung von Systemen linearer Gleichungen. Bulletin International de l’Académie Polonaise Des Sciences et Des Lettres, 35, 355–357.
![Page 21: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/21.jpg)
Randomized Kaczmarz T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
G.N. Hounsfield. Computerized transverse axial scanning (tomography): Part I. description of the system. British Journal Radiology. 1973
Karczmarz, M. S. (1937). Angenaherte Auflosung von Systemen linearer Gleichungen. Bulletin International de l’Académie Polonaise Des Sciences et Des Lettres, 35, 355–357.
![Page 22: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/22.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
![Page 23: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/23.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
![Page 24: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/24.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
![Page 25: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/25.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
![Page 26: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/26.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
Observation:
![Page 27: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/27.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
Observation:
![Page 28: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/28.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
Observation:
![Page 29: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/29.jpg)
Randomized Coordinate DescentLeventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
Observation:
Block Coord. Descent
![Page 30: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/30.jpg)
Modern Sketching
![Page 31: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/31.jpg)
Randomized Sketching
The Sketching Matrix
David P. Woodruff (2014), Foundations and Trends® in Theoretical Computer, Sketching as a Tool for Numerical Linear Algebra.
W. B. Johnson and J. Lindenstrauss (1984). Contemporary Mathematics, 26, Extensions of Lipschitz mappings into a Hilbert space.
![Page 32: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/32.jpg)
Sketching and Projecting
![Page 33: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/33.jpg)
1. Relaxation Viewpoint“Sketch and Project”
![Page 34: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/34.jpg)
1. Relaxation Viewpoint“Sketch and Project”
![Page 35: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/35.jpg)
1. Relaxation Viewpoint“Sketch and Project”
![Page 36: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/36.jpg)
2. Optimization Viewpoint “Constrain and Approximate”
![Page 37: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/37.jpg)
2. Optimization Viewpoint “Constrain and Approximate”
![Page 38: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/38.jpg)
2. Optimization Viewpoint “Constrain and Approximate”
![Page 39: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/39.jpg)
3. Geometric Viewpoint “Random Intersect”
![Page 40: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/40.jpg)
3. Geometric Viewpoint “Random Intersect”
![Page 41: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/41.jpg)
3. Geometric Viewpoint “Random Intersect”
![Page 42: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/42.jpg)
3. Geometric Viewpoint “Random Intersect”
![Page 43: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/43.jpg)
3. Geometric Viewpoint “Random Intersect”
![Page 44: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/44.jpg)
3. Geometric Viewpoint “Random Intersect”
![Page 45: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/45.jpg)
3. Geometric Viewpoint “Random Intersect”
(1)
![Page 46: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/46.jpg)
3. Geometric Viewpoint “Random Intersect”
(2)
(1)
![Page 47: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/47.jpg)
4. Algebraic Viewpoint“Random Update”
Random Update Vector
![Page 48: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/48.jpg)
4. Algebraic Viewpoint“Random Update”
Moore-Penrose pseudo inverse
Random Update Vector
Fact:
![Page 49: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/49.jpg)
4. Algebraic Viewpoint“Random Update”
Moore-Penrose pseudo inverse
Random Update Vector
Fact:
Small matrix
![Page 50: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/50.jpg)
5. Analytic Viewpoint“Random Fixed Point”
![Page 51: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/51.jpg)
5. Analytic Viewpoint“Random Fixed Point”
![Page 52: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/52.jpg)
5. Analytic Viewpoint“Random Fixed Point”
Random Iteration Matrix
![Page 53: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/53.jpg)
Theory
![Page 54: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/54.jpg)
Complexity / Convergence
Theorem [GR‘15]
![Page 55: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/55.jpg)
Complexity / Convergence
Theorem [GR‘15]
Smallest nonzero eigenvalue
![Page 56: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/56.jpg)
Case study of
![Page 57: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/57.jpg)
Case study of
Special Choice of Parameters
![Page 58: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/58.jpg)
Case study of
Special Choice of Parameters
![Page 59: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/59.jpg)
Case study of
Special Choice of Parameters
![Page 60: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/60.jpg)
Case study of
Special Choice of Parameters
![Page 61: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/61.jpg)
Case study of
Special Choice of Parameters
No zero rows in A is positive definite
![Page 62: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/62.jpg)
The rate: lower and upper bounds
Theorem [RG‘15]
![Page 63: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/63.jpg)
The rate: lower and upper bounds
Theorem [RG‘15]
Insight: The method is a contraction (without any assumptions on S whatsoever). That is, things can not get worse.
![Page 64: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/64.jpg)
The rate: lower and upper bounds
Theorem [RG‘15]
Insight: The method is a contraction (without any assumptions on S whatsoever). That is, things can not get worse.
![Page 65: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/65.jpg)
The rate: lower and upper bounds
Theorem [RG‘15]
Insight: The method is a contraction (without any assumptions on S whatsoever). That is, things can not get worse.
Insight: lower rank of A and great rank of STA gives better lower bound. In other words, when the dimension of the search space in the “constrain and approximate” viewpoint grows.
![Page 66: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/66.jpg)
Special Case: Randomized
Kaczmarz Method
T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
T. Strohmer and R. J. Vershynin, (2009). A Randomized Kaczmarz Algorithm with Exponential Convergence Journal of Fourier Analysis and Applications, 15:262
![Page 67: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/67.jpg)
Randomized Kaczmarz: derivation and rate
General Method
![Page 68: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/68.jpg)
Randomized Kaczmarz: derivation and rate
General Method
Special Choice of Parameters
![Page 69: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/69.jpg)
Randomized Kaczmarz: derivation and rate
General Method
Special Choice of Parameters
![Page 70: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/70.jpg)
Randomized Kaczmarz: derivation and rate
General Method
Special Choice of Parameters
![Page 71: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/71.jpg)
Randomized Kaczmarz: derivation and rate
General Method
Special Choice of Parameters
![Page 72: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/72.jpg)
Randomized Kaczmarz: derivation and rate
General Method
Special Choice of Parameters
![Page 73: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/73.jpg)
Randomized Kaczmarz: derivation and rate
General Method
Special Choice of Parameters
Complexity Rate.
![Page 74: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/74.jpg)
Special Case: Randomized
Coordinate Descent
T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
Leventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
![Page 75: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/75.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
![Page 76: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/76.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
Special Choice of Parameters
positive definite
![Page 77: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/77.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
Special Choice of Parameters
positive definite
![Page 78: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/78.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
Special Choice of Parameters
positive definite
![Page 79: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/79.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
Special Choice of Parameters
positive definite
![Page 80: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/80.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
Special Choice of Parameters
positive definite
![Page 81: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/81.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
Special Choice of Parameters
Complexity Rate
positive definite
![Page 82: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/82.jpg)
Theory recovers known and new convergence results
Method Convergence Rate
Randomized CDLeast square
B S
T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
*Leventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
Gaussian psd
Gaussian Kaczmarz
![Page 83: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/83.jpg)
Designing New Methods
![Page 84: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/84.jpg)
Optimal methodsOptimal choice
![Page 85: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/85.jpg)
Optimal methodsOptimal choice
B
![Page 86: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/86.jpg)
Optimal methodsOptimal choice
B
Optimal S
![Page 87: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/87.jpg)
Optimal methodsOptimal choice
B
Optimal S
S with fixed range
![Page 88: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/88.jpg)
Optimal methodsOptimal choice
B
Optimal S
S with fixed range Optimal pi's
![Page 89: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/89.jpg)
Optimal methodsOptimal choice
B
Optimal S
S with fixed range Optimal pi's Difficult SDP
![Page 90: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/90.jpg)
Practical New MethodsOne Shot Sketches
T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
N. Ailon and B. Chazelle (2006). Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform. Mathematics of Operations Research, 35(3), 641–654.
![Page 91: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/91.jpg)
Practical New MethodsOne Shot Sketches
S
Gaussian Matrix
T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
N. Ailon and B. Chazelle (2006). Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform. Mathematics of Operations Research, 35(3), 641–654.
Subsampled Hadamard-Welsh
Countmin Sketch
Computing STA
![Page 92: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/92.jpg)
Practical New MethodsOne Shot Sketches
S
Gaussian Matrix
T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
N. Ailon and B. Chazelle (2006). Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform. Mathematics of Operations Research, 35(3), 641–654.
Subsampled Hadamard-Welsh
Countmin Sketch
Computing STA
Rademacher Sketch
![Page 93: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/93.jpg)
Sub-Rademacher Sketching
![Page 94: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/94.jpg)
1
1
Sub-Rademacher Sketching
![Page 95: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/95.jpg)
1
1
2
2
Sub-Rademacher Sketching
![Page 96: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/96.jpg)
1
1
2
23
3
1
3
2
Sub-Rademacher Sketching
![Page 97: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/97.jpg)
1
1
2
23
3
1
3
2
Sub-Rademacher Sketching Flip the sign with 50% probability
![Page 98: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/98.jpg)
1
1
2
23
3
1
3
2
Sub-Rademacher Sketching Flip the sign with 50% probability
![Page 99: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/99.jpg)
Experiments
![Page 100: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/100.jpg)
Large scale Ridge Regression
Problem: w8a
Conjugate Gradient
Block Coordinate
Descent
Rademacher Sketch?
Origin: LIBSVM
![Page 101: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/101.jpg)
Large scale Ridge Regression
Problem: rcv1
Conjugate Gradient
Block Coordinate
Descent
Rademacher Sketch?
Origin: LIBSVM
![Page 102: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/102.jpg)
Large scale Ridge Regression
Problem: mnist
Conjugate Gradient
Origin: LIBSVM
![Page 103: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/103.jpg)
Conclusions
Unites many randomized methods under a single framework
Improved convergence New lower bounds, less assumptions, tightest results.
Design new methods S = Guassian, count-sketch, Walsh-Hadamard ...etc
Optimal Sampling We can choose a sampling that optimizes the convergence rate.
![Page 104: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/104.jpg)
Large scale Ridge Regression
Problem: a9a
Origin: LIBSVM
Conjugate Gradient
Rademacher Sketch
Block Coordinate
Descent
![Page 105: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/105.jpg)
Large scale Ridge Regression
Problem: a9a
Origin: LIBSVM
Conjugate Gradient
Rademacher Sketch
Block Coordinate
Descent
Fast initial sublinear convergence
![Page 106: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/106.jpg)
RMG and Peter RichtárikStochastic Dual Ascent for Solving Linear SystemsPreprint arXiv:1512.06890, 2015
RMG and Peter RichtárikRandomized quasi-Newton updates are linearly convergent matrix inversion algorithmsPreprint arXiv:1602.01768, 2016
RMG and Peter RichtárikRandomized Iterative Methods for Linear Systems. SIAM. J. Matrix Anal. & Appl., 36(4), 1660–1690, 2015. Most Downloaded SIMAX Paper!
![Page 107: Randomized iterative methods for linear systems · 2020-05-13 · Linear Systems Preprint arXiv:1512.06890, 2015 RMG and Peter Richtárik Randomized quasi-Newton updates are linearly](https://reader033.fdocuments.in/reader033/viewer/2022052717/5f03d9927e708231d40b11ee/html5/thumbnails/107.jpg)
Thank you,Questions?