Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the...
Transcript of Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the...
![Page 1: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/1.jpg)
Randomized Kaczmarz
Nick Freris EPFL
(Joint work with A. Zouzias)
![Page 2: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/2.jpg)
Outline
▪ Randomized Kaczmarz algorithm for linear systems • Consistent (noiseless) • Inconsistent (noisy)
▪ Optimal de-noising • Convergence analysis and simulations
▪ Application in sensor networks • Distributed consensus algorithm for synchronization
▪ Faster convergence and energy savings
- Faster for sparse systems - Consensus design method
1 / 17
![Page 3: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/3.jpg)
Applications
▪ Computer science • Parallel and distributed algorithms • Random projections
▪ Sensor networks • Optimization & Control • Distributed estimation • Consensus
▪ Signal processing • Sampling • Compressed Sensing • Linear Inverse problems
▪ Imaging (ART) ▪ Tomography ▪ Acoustics ▪ and more..
2 / 17
Convergence lab (CSL, Univ. Illinois)
SmartSense, EPFL
![Page 4: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/4.jpg)
Kaczmarz algorithm ▪ Iterative algorithm for solving
• also known as ART in image reconstruction / tomography
▪ Convergece: alternating projections
performance depends on row order
Projection to the solution space of selected row
Round-Robin row selection
3 / 17
Ax = b
![Page 5: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/5.jpg)
Randomized Kaczmarz ▪ Iterative algorithm for solving
▪ Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence:
Projection to the solution space of selected row
Randomized selection of row
4 / 17
2F :=
||A||2F�2min
Ax = b
performance depends on row scaling
![Page 6: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/6.jpg)
Noisy case ▪ Noisy measurements: ▪ Oscillatory behavior
• Asymptotically constrained in a ball (N’10, FZ’12)
▪ Under-relaxation (RKU)
• Convergence to a point in the ball ▪ slower
▪ Least-squares: • Bad idea (squaring the condition number)
5 / 17
![Page 7: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/7.jpg)
Optimal de-noising ▪ LS for inconsistent system:
• Solution: projection to the range space of A
Ax = bR(A)
Projection to the orthogonal complement of the selected column
Randomized selection of column
6 / 17
same rate of convergence
![Page 8: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/8.jpg)
Putting the pieces together
Randomized orthogonal projection
Randomized Kaczmarz
▪ RK and de-noising:
7 / 17
Termination criteria
![Page 9: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/9.jpg)
Analysis of REK ▪ Rate of convergence (ZF’13):
▪ same exponent, no delay
▪ Expected number of arithmetic operations:
• proportional to ▪ sparsity ▪ squared condition number
8 / 17
Ekx(k) � xLSk2 (1� 1
2F (A)
)k[kxLSk2 + ckbR(A)k2k]
Designed for sparse well-conditioned systems
![Page 10: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/10.jpg)
Implementation ▪ Implementation in C
• REK-C • REK-BLAS (level-1 BLAS routines + Blendenpik)
▪ Comparison • Matlab backslash \ • LAPACK
▪ DGELSY (QR factorization) ▪ DGELSD (SVD) ▪ LSNR
• Blendenpik
9 / 17
![Page 11: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/11.jpg)
Experiments
Excellent performance for sparse systems
10 / 17
![Page 12: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/12.jpg)
A sensor network problem
▪ Relative measurements • For two neighbors: • Network problem:
▪ Jacobi algorithm for LSE
• Local averaging (distributed)
▪ Synchronous: Exponential convergence (GK’06) ▪ Asynchronous: Exponential convergence (FZ’13)
▪ Applications • Clock synchronization (smoothing time differences) • Localization (smoothing distance/angular differences)
11 / 17
yij = xi � xj + wij
![Page 13: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/13.jpg)
Smoothing via RK
▪ Asynchronous implementation • Exponential clocks
Distributed averaging
12 / 17
Randomized sampling
![Page 14: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/14.jpg)
An extension
▪ “Over-smoothing” (RKO)
•
Faster convergence in absolute time vs
More messages / iteration
13 / 17
![Page 15: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/15.jpg)
Convergence analysis Algorithm Convergence Reference
Jacobi GK’06 (FZ’12)
OSE Faster than Jacobi BDE’06
RKS FZ’12
RKLS FZ’12
RKU FZ’12
RKO Faster than RKS FZ’12
▪ Cheeger’s inequality:
14 / 17
depends on network connectivity
O(�2(G)
m)
O(�2(G)
m)
O(�2(G)
m)
O(�22(G)
m2)
![Page 16: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/16.jpg)
Simulations
Faster convergence Energy savings
15 / 17
![Page 17: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/17.jpg)
Conclusions ▪ Randomized Kaczmarz (RK) algorithm
• Exponential convergence in the mean-square ▪ Same rate regardless of noise
• Distributed asynchronous smoothing
▪ Experiments • Linear systems: Gains for sparse systems • Sensor networks: Faster convergence and energy savings
Efficient sparse linear system solver
16 / 17
![Page 18: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/18.jpg)
Ongoing work
▪ Distributed implementation of REK • Range projection • matrix pre-conditioning • termination criteria
▪ Stochastic approximation • convergence to the true values
▪ slower (gradient method) • improved convergence
17 / 17
Numerical analysis is not dead!
![Page 19: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/19.jpg)
References
1. N. Freris and A. Zouzias, “Fast distributed smoothing of relative measurements," 51st
IEEE Conference on Decision and Control (CDC), pp.1411-1416, 10-13 Dec. 2012.
2. Anastasios Zouzias and Nikolaos Freris, “Randomized Extended Kaczmarz for Solving Least Squares.” SIAM Journal on Matrix Analysis and Applications, 34(2), 773-793, 2013.
3. T. Strohmer and R. Vershynin, “A Randomized Kaczmarz Algorithm with Exponential
Convergence,” Journal of Fourier Analysis and Applications, vol. 15, no. 1, pp. 262–278, 2009.
4. D. Needell. “Randomized Kaczmarz Solver for Noisy Linear Systems.” Bit Numerical Mathematics, 50(2):395–403, 2010.
![Page 20: Randomized Kaczmarz - EPFL · Randomized Kaczmarz (RK) algorithm • Exponential convergence in the mean-square Same rate regardless of noise • Distributed asynchronous smoothing](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0cf1fa7e708231d437e873/html5/thumbnails/20.jpg)
Thank you