Statistical Physics of Inference - Bienvenue · information theory machine learning Tuesday, ......
Transcript of Statistical Physics of Inference - Bienvenue · information theory machine learning Tuesday, ......
My research
Physics
signal processing
optimization
information theory
machine learning
Tuesday, December 9, 14
An example of a success story:
Compressed sensingwith F. Krzakala, M. Mezard, F. Sausset, Y. Sun,
Tuesday, December 9, 14
From 106 wavelet coefficients, keep 25.000
Why do we record a huge amount of data, and then keep only the important bits?
Couldn’t we record only the relevant information directly?
Most signal of interest are sparse in an appropriated basis⇒Exploited for data compression (JPEG2000).
Tuesday, December 9, 14
Possible applications- Rapid Magnetic Resonance Imaging- Rapid Computed Tomography- Image acquisition (single-pixel camera)- Detection of rare genes with small number of tests- Enhance resolution in very noisy devices- .... many more .....
Left: image acquired with acceleration by a factor 2.5
Lustig, Donoho, Pauly ’07
Tuesday, December 9, 14
Setting of the problemDesign the matrix F such that sparse signal x can be reconstructed efficiently from measurements y.
= F
y
x
yµ =NX
i=1
Fµixi MN
x is sparse, i.e. only elements are non-zero. The linear problem has many solutions, only is one sparse.
⇢N
Tuesday, December 9, 14
↵=
M N
⇢ =K
N
N-component signal, K of them nonzero, M measurements,
N ! 1
impossible
easy with linear programing
?
State-of-the-art in 2011
Tuesday, December 9, 14
Statistical physics formulation
P (~x|~y) = 1
Z
e
PNi=1 log [(1�⇢)�(xi)+⇢�(xi)]
e
PMµ=1
12�µ (yµ�
PNi=1 Fµixi)2
local magnetic field
spin-variablesxi
N-body interactions
Z: partition function
Optimal signal reconstruction = equilibrium solution.Methods from physics of mean field spin glasses: replica method, cavity method, message passing. 40 years of work in physics (Mezard, Parisi, Zecchina, Nishimori ....)
Boltzmann distribution
Tuesday, December 9, 14
1st result of statistical physics analysis:
αL1αBEPα=ρ
0.4 0.5 0.6 0.7 0.8 0.90
0.05
0.1
0.15
0.2
α
Mea
n s
quar
e er
ror
L1
BEP
1 10-5 0.0001 0.001 0.01 0.1
-1
-0.5
0
0.5
1
Mean square error
tanh[4
!(E
)]
α = 0.8
α = 0.6
α = 0.5
α = 0.3
αL1αrBPα=ρ
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
ρ
α
αL1(ρ)αrBP(ρ)S-BEP
α = ρ
0.4 0.5 0.6 0.7 0.8 0.9
30
100
300
1000
3000
10000
α
Num
ber
of iter
atio
ns
rBP
Seeded BEP - L=10
Seeded BEP - L=40
fraction of non-zeros
mea
sure
men
ts p
er s
igna
l ele
men
t
‣ Better performance for a class of signals.‣ Algorithmic barrier = spinodal of a 1st order phase
transition. Algorithms blocked in a metastable state.
Tuesday, December 9, 14
Heating pad or hand warmer:
sodium acetate melts at 58 C
Thanks to: UCGP 2008, Kyoto, Japan
Nucleation for optimality!
Tuesday, December 9, 14
Implementing nucleation in compressed sensing:
0
BBBB@
1
CCCCA= ⇥
y F s
: unit coupling
: no coupling (null elements)
: coupling J1: coupling J2
J1J1
J1J1
J1J1
J1
J2J2
J2J2
J2J2
11
11
11
1
1 J2
0
0
0
BBBB@
1
CCCCA
0
BBBBBBBBBBBB@
1
CCCCCCCCCCCCA
Tuesday, December 9, 14
Thanks to statistical physics analysis compressed sensing is today computationally tractable down the information theoretic limit! Krzakala, Mezard, Sausset, Sun,
Zdeborova, Phys. Rev. X 2012. Proof: Donoho, Javanmard, Montanari, ISIT 2012.
αL1αBEPα=ρ
0.4 0.5 0.6 0.7 0.8 0.90
0.05
0.1
0.15
0.2
α
Mea
n s
quar
e er
ror
L1
BEP
1 10-5 0.0001 0.001 0.01 0.1
-1
-0.5
0
0.5
1
Mean square error
tanh[4
!(E
)]
α = 0.8
α = 0.6
α = 0.5
α = 0.3
αL1αrBPα=ρ
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
ρ
α
αL1(ρ)αrBP(ρ)S-BEP
α = ρ
0.4 0.5 0.6 0.7 0.8 0.9
30
100
300
1000
3000
10000
α
Num
ber
of iter
atio
ns
rBP
Seeded BEP - L=10
Seeded BEP - L=40
Tuesday, December 9, 14
L1
BEP
S-BEP
α = 0.5 α = 0.4 α = 0.3 α = 0.2 α = 0.1
α = ρ ! 0.15
Shepp-Logan phantom, sparse in the Haar-wavelet representation
Tuesday, December 9, 14
(1) Pick an important problem. (2) Compute phase transitions in an amenable setting. (3) Use gained insight to develop better algorithms.
Statistical Physics of Inference
Our cook-book:
Our main contribution: Algorithms for practitioners. Conjectures for mathematicians.
Tuesday, December 9, 14
Read more: F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Statistical physics-based reconstruction in compressed sensing, Phys. Rev. X (2012).
F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices, J. Stat. Mech. (2012).
Another example of this approach:F. Krzakala, E. Mossel, C. Moore, J. Neeman, A.Sly, LZ, P. Zhang, Spectral Redemption in Clustering Sparse Networks, PNAS (2013).
A. Decelle, F. Krzakala, C. Moore, LZ, Inference and phase transitions in detection of modules in sparse networks, Phys. Rev.lett. (2011)
Tuesday, December 9, 14
Thank you for your attention!
Read more: F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Statistical physics-based reconstruction in compressed sensing, Phys. Rev. X (2012).
F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices, J. Stat. Mech. (2012).
Another example of this approach:F. Krzakala, E. Mossel, C. Moore, J. Neeman, A.Sly, LZ, P. Zhang, Spectral Redemption in Clustering Sparse Networks, PNAS (2013).
A. Decelle, F. Krzakala, C. Moore, LZ, Inference and phase transitions in detection of modules in sparse networks, Phys. Rev.lett. (2011)
Tuesday, December 9, 14