Post on 26-Sep-2020
Computational Neuroscience ESPRC Workshop--Warwick
Information-GeometricStudies on Neuronal SpikeTrains
Shun-ichi AmariShun-ichi AmariRIKEN Brain Science Institute
Mathematical Neuroscience Unit
Neural Firing
1x 2x3
xn
x
higher-order correlations
orthogonal decomposition
1 2( ) ( , ,..., ) : joint probabilityn
p p x x x=x
[ ]i ir E x=
[ , ]ij i jv Cov x x=
----firing rate
----covariance: correlation
1,0i
x =
1 2{ ( , ,..., )}n
S p x x x=
( ) 0,1i
x t =
time tneuron i
1 T
1 ( )11x
1( )x T
( )21x ( )2
x T
n ( )1n
x ( )nx T
Multiple spike sequence:
{ ( ), 1,..., ; 1,..., }i
x t i n t T= =
( ){ } ( )( )
2
2
1; , ; , exp
22
xS p x p x
μμ μ= =
Information GeometryInformation Geometry ? ?
( ){ }p x
μ
( ){ };S p x= è
Riemannian metric
Dual affine connections
( , )μ=è
Information GeometryInformation Geometry
Systems Theory Information Theory
Statistics Neural Networks
Combinatorics PhysicsInformation Sciences
Riemannian Manifold
Dual Affine Connections
Manifold of Probability Distributions
Math. AI
Manifold of Probability DistributionsManifold of Probability Distributions
( )1 2 3 1 2 3
1, 2,3 { ( )}
, , 1
x p x
p p p p p p
=
= + + =
3p
2p
1p
p
( ){ };M p x=
Manifold of Probability DistributionsManifold of Probability Distributions
( ){ },
1, 2,3
P x
x
=
=
M p
3
21
( ) ( ) ( )3
3 1 2
1
, 1i i
i
P x p x p p p=
= =p
3
21
M
( )1 2 3
1 2 3
, ,
1
p p p
p p p
=
+ + =
p
2 2 2
1 2 31
i ip=
+ + =
( )
11
3 1 2
22
3
log
,
log
p
p
p
p
=
=
=
InvarianceInvariance ( ){ },S p x=
1. Invariant under reparameterization1. Invariant under reparameterization
( )( , ) ,p x p x=
2. 2. Invariant under different representationInvariant under different representation
( ) ( ), { , } { ( , )}y y x p y p x= = ( ) ( )2
1 2
2
1 2
, ,
| ( , ) ( , ) |
p x p x dx
p y p y dy
2 2
i iD =
Two StructuresTwo Structures
Riemannian metricRiemannian metric
affine connection --- geodesicaffine connection --- geodesic
( )2,ij i jds g d d d d= =< >
Fisher informationFisher information
log logij
i j
g E p p=
Affine Connection
covariant derivative
geodesic X=X X=X(t)
( )
c
i j
ij
X Y
s g d d
=
=
minimal distance
& &
straight line
Affine Connectioncovariant derivative; parallel transport
,
geodesic =x x=x(t)
:
, ,
( )
X c
i j
i
i j
ij
j
Y X Y
x
s g d
X Y X
d
Y g X Y< >=< >
=
=
minimal distance straight line
& &
Duality: two affine connectionsDuality: two affine connections
, , ,i j
ijX Y X Y X Y g X Y= < >=
Riemannian geometry:Riemannian geometry: =
X
Y
X
Y
*
{ , , , *}S g
Dual Affine ConnectionsDual Affine Connections
e-geodesice-geodesic
m-geodesicm-geodesic
( ) ( ) ( ) ( ) ( )log , log 1 lgr x t t p x t q x c t= + +
( ) ( ) ( ) ( ), 1r x t tp x t q x= +
( ),
( )q x
( )p x
*( , )
( ) 0
* ( ) 0
x
x
x t
x t
=
=
&
&
&
&
Information GeometryInformation Geometry
-- Dually Flat Manifold-- Dually Flat Manifold
Convex Analysis
Legendre transformation
Divergence
Pythagorean theorem
I-projection
Dually Flat ManifoldDually Flat Manifold
1. Potential Functions1. Potential Functions
---convex (Bregman, Legendre transformation)
2. Divergence2. Divergence [ ]:D p q
3. Pythagoras Theorem3. Pythagoras Theorem
[ ] [ ] [ ]: : :D p q D q r D p r+ =
4.4. Projection TheoremProjection Theorem
5.Dual foliation5.Dual foliation
p
r
q
Pythagoras Theorem
p
q
r
D[p:r] = D[p:q]+D[q:r]
p,q: same marginals
r,q: same correlations
1 2, independent
correlations
( )[ : ] ( ) log
( )x
p xD p r p x
q x=
estimation correlationtesting
invariant under firing rates
Projection Theorem
[ ]min :Q M
D P Q
Q = m-geodesic
projection of P to M
[ ]min :Q M
D Q P
Q = e-geodesic projection of P to M
( ) 0,1i
x t =
time tneuron i
1 T
1 ( )11x
1( )x T
( )21x ( )2
x T
n ( )1n
x ( )nx T
Multiple spike sequence:
{ ( ), 1,..., ; 1,..., }i
x t i n t T= =
spatial correlationsspatial correlations
( )1, ,
nx x= Lx
( ) ( )1 1exp
i i ij i j n n
i j
p x x x x x<
= + + +L
LLx
[ ] { }
{ }Prob 1
Prob 1
i i i
ij i j i j
r E x x
r E x x x x
= = =
= = = =
( )
( )1
1
, , ,
, , ,
i ij n
i ij nr r r
=
=
L
L
L
Lr
L
orthogonal structure
( ){ }S p= x : coordinates
Spatio-temporal correlationsSpatio-temporal correlations
correlated Poisson
( )p x : temporally independent
correlated renewal process
( )tp x : firing rate ( )ir t modified
spatial correlations fixed
firing rates:
correlation—covariance?
1x
2x
3x
00 01 10 11{ , , , }p p p p
1 2 12, ;r r r
Two neurons:
Correlations of Neural Firing
( ){ }{ }
1 2
00 10 01 11
1 1 10 11
2 1 01 11
,
, , ,
p x x
p p p p
r p p p
r p p p
= = +
= = +
11 00
10 01
logp p
p p=
1x 2
x
2
1
1 2{( , ), }r r
orthogonal coordinates
firing rates
correlations
1 2{ ( , )}S p x x=
1 2, 0,1x x =
1 2{ ( ) ( )}M q x q x=
Independent Distributions
Orthogonal Coordinates:
firing rates
r
correlation fixedscale of correlation
r’
two neuron casetwo neuron case
( )
( )( )
( )
( ) ( ) ( )( )
1 2 12 1 2 12
12 12 1 200 11
12
01 10 1 12 2 12
12 1 2
12 1 2
, , ; , ,
1log log
, ,
, ,
r r r
r r r rp p
p p r r r r
r f r r
r t f r t r t
+= =
=
=
Decomposition of KL-divergence
D[p:r] = D[p:q]+D[q:r]
p,q: same marginals
r,q: same correlations
1 2,
p
q
rindependent
correlations
( )[ : ] ( ) log
( )x
p xD p r p x
q x=
pairwise correlations
ij ij i jc r r r=
independent distributions
,,ij i j ijk i j kr r r r r r r= = L
How to generate correlated spikes?
(Niebur, Neural Computation [2007])
higher-order correlations
covariance: not orthogonal
Orthogonalhigher-order correlations
( )
( )1
1
;
;
,
,
,
,
i i n
i n
j
i jrr r
=
=
L
L
L
Lr
tractable models ofdistributions
Full model: parameters
Homogeneous model: n parameters
Boltzmann machine: only pairwise
correlations
Mixture model
2 1n
{ ( )}M p= x
Models of Correlated Spike Distributions (1)
Reference 0110001011 ( )y t
( )1x t%
( )2x t%
( )3x t%
1001011001
0101101101
Models of Correlated Spike Distributions (2)
( ) ( ) ( )i ix t x t y t= % o
additive interaction
eliminating interaction
Niebur replacement
( )y t
( )1
2 ( )
x t
x t
%
%
0110001011
1001001101
1100010011
Mixture Model
( ):y t 0110001011 mixing sequence
( ) ( )1; , when 1
i ip p x u y= =x u
( ) ( )2; , when 0
i ip p x v y= =x u
( , ) : 1 with probability p x u x u=
( ) ( ) ( | )p p y p y=x x
Mixture modelMixture model
( ) ( )
( ) ( ) ( )
; ,
; , , ; ;
i ip p x u
p m mp mp
=
= +
x u
x u v x u x v
( )
( )1
1
, , 1
, ,
n
n
u u m m
v v
= =
=
L
L
u
v
( ){ } ( ){ }; , , ,n n
M p m S p= =x u v x
: independent distribution
firing ratesfiring rates
ij i j i jr mu u mv v= +
ijk i j k i j kr mu u u mv v v= +
i i ir mu mv= +
L
Conditional distributionsand mixture
1,2,3,...
( | ) : Hebb assembly
( ) ( ) ( | )
y
p y y
p p y p y
=
=
x
x x
State transition of Hebb assembliesmixture and covariances
( ) ( ) ( ) ( ), 1u v
p t t p tp= +x x x
( ) ( ) ( ) ( ) ( )( )( )1 1ij ij ij i i j j
c t t c u tc v t t u v u v= + +
Extra increase (decrease) of covariances
within Hebb assemblis---- increase between Hebb assemblies ---decrease
Temporal correlations {x(t)}, t =1, 2, …T
Poisson sequence:
Markov chain
Renewal process
Prob{ (1),..., ( )}x x T
Orthogonal parameter ofcorrelation ---- Markov case
1
1
1 1
( | )
Pr{ | }
Pr{ } Pr( ) ( | )
t t
ij t t
t t t
p x x
a x i x j
x x p x x
+
+
+
= = =
=
1
2
11 1
11 00
10 01
Pr{ 1}
log
tr x
c r r
a a
a a
= =
=
=
Spatio-temporal correlation
correlated Poisson
1({ }) ( )
T
t tt
p p=
=x x
Correlated renewal process
,
Pr( ) ( ') : ' last spike time
Pr( ) ( ') ( | )
t
i t i
x k t t t
x k t t p x
=
= x
Neurons
1x
nx
( )1i i
x u=
Gaussian [ ]i i j
u E u u= =
2x
Population and Synfire
Population and Synfire
hswu jiji =
[ ]ii
ux 1=
(1 )i i
u h= +
( ), 0, 1i
N
s
1x
nx
2
[ ]
[ ] 1
i j
i
E u u
E u
=
=
{ } timesame at the fire neurons Prob ipi=
Pr{ neurons fire}r
ir P nr
n= =
( ) ( )( , )
nH r nzq r e e d=
( ) ( ) ( )FrFrn
z = 1 log 1 log 2
2
( ) dt 2
1 2
0
2t
ha
ehaFF ==
1 22 1( , ) exp[ { ( ) } ]
2(1 ) 2 1q r c F h=
1 2
1
...
( , ) exp{ ...}
(1/ )k
i i ij i j ijk i j k
k
i i i
p x x x x x x
O n
= + + +
=
x
Synfiring
( )
1( ) ( ,..., )
1
n
i
p p x x
r x q rn
=
=
x
( )q r
r
Bifurcation
r
rP
ix : independent---single delta peak
pairwise correlated
higher-order correlation
Semiparametric StatisticsSemiparametric Statistics
( ){ }( ) ( )
, ,
,
M p x r
p x r x
=
=
y x=
'
i i i
i i i
y
x
= +
= +
mle, least squre, total least square
( ) ( ) ( ), ; , ; ,p x y p x y r d=
x
y
Linear Regression: Semiparametrics
( )
( )
( )
1 1
2 2
,
,
,n n
x y
x y
x y
M
'
i i i
i i i
x
y
= +
= +
( )' 2, 0,
i iN
y x=
Statistical Model
( ) ( ) ( )
( )
( ) ( ) ( )
2 2
1
1 1, , exp
2 2
, , : , , ,
, , ,
i i i n
p x y c x y
p x y
p x y p x y Z d
=
=
L
semiparametric
Least squares?
( ) ( )
( )( )
2
2ˆmin :
1,
mle, TLS
0
Neyman-Scott
i i
i i
i
ii
i i
i i i i
x yL y x
x
yy
n x x
y x y x
= =
+ =
( )
( ) ( )
( )
( )
'
1 2
,
,
, , , ,
, , 0
, 0
ˆ, 0
Z
Z
i
x x p x Z
y x E y x
E y x
y x
=
=
Semiparametric statistical model
Estimating function
Estimating equation
L
( )
( ) ( )
( ){ }( )
( )( )
, ; , log
, ; ,
, ;
, ,I
u x y Z p
sE s
v x y Z E f s
k s x y
u x y u E u s
k x y y x
=
=
=
=
=
= +
L
Fibre bundreFibre bundre
function spacefunction space
r
( ) ( )( ) ( )( )
2
2
,
, ,I
z N
u x Z x y c y x
c
μ
μ
= + +
=
( ) ( )( )
( )
2
222 2
, ;
1
1
i
i
f x c x y c y x
c
xn
xn
μ
μ
μ
= + +
=
=
=
Poisson process
Poisson Process: Instantaneous firing rate is constant over
time. dtFor every small time window dt,
generate a spike with probability dt.
T
Cortical Neuron Poisson Process
Poisson processPoisson process
cannot explain inter-cannot explain inter-
spike intervalspike interval
distributions.distributions.
TeTp =)(
(Softky & Koch, 1993)
Gamma distribution
Gamma Distribution: Every -th spike of the Poisson process is left.
: Firing rate
: Irregularity.T
)(
)(),q(T;
T1= e Two parameters
=1 (Poisson)
=3
T
Gamma distributionGamma distribution
( )( )( )
{ }1exp
rf T T r T=
1=
Integrate-and fire
Markov model
: Poisson
: regular
Irregularity is unique to
individual neurons.
t
t
t
Regular large
Irregular small
Irregularity varies among
neurons.)
We assume that We assume that is independent of is independent oftime.time.
Information geometry of
estimating function•Estimating function y(T, ):
(See poster for details)
)(
),(log),(
),(log),(
îk
kT;êpkT;êv
dê
kT;êpdkT;êu
How to obtain an estimatingfunction y:
•Maximum likelihood Method:
vvv
vuuy
><
><=
Scorefunctions
E[y(T, )]=0
0);(1
==
N
l
lTy0);()()(log
1
1 ===
N
l
lN TuTpTpd
dL
temporal correlationstemporal correlations
( ) ( ) ( )1 2x x x NL
independent with firing probability ( )r t
spike counts: Poisson
ISI: exponential
renewal:
( ) ( )ir t k t t= : last
spikeISI distribution ( )f t
( ) ( ) ( )0
exp
T
f T ck T k t dt=
Estimation of
by estimating functionsNo estimating function exists if the neighboring firing rates aredifferent.
m( ) consecutive observations must have the same firing rate.
Estimating function:
(E[y]=0)
Example:Example:m=2m=2
( ))(2)2(2log
2
21
21 êöêöTT
TTy +
+=
l-thset:
( )0)(2)2(2log
1
12
21
21 =++
==
êöêöTT
TT
Ny
,m
lll
ll
=0
2121 )(),;(),;(),;,( dkTqTqkTTp
)( 21
ll,TT
Model:
Case of m=1
(spontaneous discharge)
Firing rate continuously
changes and is driven by
Gaussian noise.: Correlation
( )0)(2)2(2log
1
12
21
21 =++
==
êöêöTT
TT
Ny
,m
lll
ll
can be approximated if can be approximated ifthe firing rate changesthe firing rate changesslowly.slowly.Estimating function (m=2):
slow