Statistical Neurodynamics of Deep...
Transcript of Statistical Neurodynamics of Deep...
![Page 1: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/1.jpg)
Statistical Neurodynamicsof Deep Networks
Shun‐ichi Amari
RIKEN Brain Science Institute
![Page 2: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/2.jpg)
Statistical NeurodynamicsRozonoer (1969)Amari (1971; 197Amari et al (2013)Toyoizumi et al (2015)Poole, …, Ganguli (2016)
~ (0 , 1)ijw N
Macroscopic behaviorscommon to almost all (typical) networks
![Page 3: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/3.jpg)
Macroscopic variables
2
1
1
1activity :
distance: = [ : ']curvature :
( )( )
i
l l
l l
A xn
D D
A F AD K D
x x
![Page 4: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/4.jpg)
Deep Networks
01
2
1
( )
1
( )
i ij i il l
il ll
l l
x w x w
A xn
A F A
2
0
~ (0, 1 / )
0, 1'(0) =const
ij
i
w N n
w
![Page 5: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/5.jpg)
Pullback Metric
2 1la b
abl
ds g g dx dx d dn
x x
![Page 6: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/6.jpg)
1ab a b
l
gn
e e
![Page 7: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/7.jpg)
1l ln n
![Page 8: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/8.jpg)
Poole et al (2016)Deep neural networks
![Page 9: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/9.jpg)
Dynamics of Activity
2 20
1
20
( ) ( )~ (0, )
1 ( ) [ ( ) ] ( )
( ) ( ) ~ (0,1)
k k
l
y w y uu N A
A y E u An
A Av Dv v N
![Page 10: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/10.jpg)
0
(0) (0) 1
( )
convergei
A A
x
![Page 11: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/11.jpg)
Dynamics of Metric
2 2
21
( ) ( '( ) )
E[ '( )) ] E[ '( )) ]E[ ]
mean field approximation
( ) '( )
k k
a a
k k
ab k j kj
k j k j
dy B dyB
B B u w
g B B g
u w w u w w
A Av Dv
e e
![Page 12: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/12.jpg)
1
1 1
1
( )conformal transformation!
( )
ab ab
ab
ll
ab ab
g A g
A
g
rotation, expansion
![Page 13: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/13.jpg)
Dynamics of Curvature
2 2
''( )( )( ) '( )
| |
ab a b a b
a b a b
ab ab ab
ab ab
H y
u
H
e
w e w e w e
H H H
H
![Page 14: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/14.jpg)
22
2 21
2 1 1
12
2 1
( ) ''( )
( ) ( )(2 1) ( )1
( )(2 1)
exponwntial expansion!
l l l l l
ab abab
ll
ab ab
A Av Dv
H A A A H
H l A
![Page 15: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/15.jpg)
Dynamics of Distance (Amari, 1974)
21( , ') ( ')
1( , ') ' '
' 2
~N(0, V)
' ' V=
( ') E[ (
ii
i i
k k
k k
D x x x xn
C x x x x x xn
D A A C
u w y
u w y A C
C A C A
) ( ' )]C C A C C
![Page 16: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/16.jpg)
1
1
( )
1
l lD K D
dDdD
![Page 17: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/17.jpg)
Poole et al (2016)Deep neural networks
![Page 18: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/18.jpg)
Problem!
( , )( )
equidistance property
l lD DD K D
x x
![Page 19: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/19.jpg)
Shuttering
Multiplicity
Dynamics of recurrent net
Dropout and backprop
![Page 20: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/20.jpg)
Multilayer Perceptrons
i iy v w x
, i if v x w x
1 2( , ,..., )nx x x x
1 1( ,..., ; ,..., )m mw w v v
1 w x
yx
![Page 21: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/21.jpg)
Multilayer Perceptron
1 1,
,
, ; ,i i
m m
y f
v
v v
x θ
w x
θ w w
neuromanifold ( )x
space of functions S
![Page 22: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/22.jpg)
singularities
![Page 23: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/23.jpg)
Geometry of singular model
y v n w x
W
vv | | 0w
![Page 24: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/24.jpg)
1 , ,
:
t t t tG y
G l l
Fisher
Natural Gradient Stochastic Descent
Information Matrix
invarint; steepest descent
x
![Page 25: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/25.jpg)
model: 2 hidden neurons
2
1 1 2 2
2
,
,
12
tu
f w w
y f
u e dt
x J x J x
x
![Page 26: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/26.jpg)
Singular Region in Parameter Space
1 2 1 2
1 2 2
1 2 1
1 1 2 2
, ,
0, ,
, 0,
,
R w w w w
w w w
w w w
f w w
J J J J
J J
J J
x J x J x
![Page 27: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/27.jpg)
Coordinate transformation
1 1 2 2
1 2
1 2
2 1
2 1
1 2
,
,
,
, , ,
w ww w
w w w
w wzw w
w z
J Jv
u J J
v u
![Page 28: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/28.jpg)
Singular Region , 0 1R w z J u
![Page 29: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/29.jpg)
Milnor attractor
![Page 30: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/30.jpg)
Topology of singular R
2 21
2 32
blow-down coordinates , ,
1 ,
1 ,
, 1n
c z u u
c z z u
S
: = e
u
ue eu
![Page 31: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/31.jpg)
Dynamic vector fields: Redundant case
![Page 32: Statistical Neurodynamics of Deep Networkskabuto.phys.sci.osaka-u.ac.jp/~koji/workshop/slides/deep/...Statistical Neurodynamics Rozonoer(1969) Amari (1971; 197 Amari et al (2013)](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f7460bd25a1e07dee1d0a1a/html5/thumbnails/32.jpg)