LNP cascade model

!"#$%&'("!)$&

*+"#)'#+#!"#$%&"),

-+"..+#'.*"/"#0

• Simplest successful descriptive spiking model

• Easily fit to (extracellular) data

• Descriptive, and interpretable (although not mechanistic)

For a Poisson model, response is captured by relationship between the distribution of red points (spiking stim) and blue points (raw stim), expressed in terms of Bayes’ rule:

s1

s2

P(spike|stim) =P(spike, stim)

P(stim)

This cannot be estimated directly...

ML estimation of LNP

[on board]

ML estimation of LNPfθ(!k · !x)If is convex (in argument and theta),

and log is concave, the likelihood of the LNP model is convex(for all observed data, )

fθ(!k · !x)

{n(t), !x(t)}

[Paninski, ’04]

ML estimation of LNPfθ(!k · !x)If is convex (in argument and theta),

and log is concave, the likelihood of the LNP model is convex(for all observed data, )

fθ(!k · !x)

{n(t), !x(t)}

[Paninski, ’04]

Examples: e(!k·!x(t))

(!k · !x(t))α, 1 < α < 2

Simple LNP fitting

• Assuming:

- stochastic stimuli, spherically distributed

- mean of spike-triggered ensemble is shifted from that of raw ensemble

• Reverse correlation gives an unbiased estimate of k (for any f).

• For exponential f, this is the ML estimate!

- Bussgang 52; de Boer & Kuyper 68

Computing the STA

s1

s2raw stimuli

spiking stimuli

STA corresponds to a “direction” in stimulus space

Projecting onto the STA

P(

spike(t) | !k · !s(t))

= P(

spike(t) & !k · !s(t))

/P (!s(t))

Solving for nonlinearity nonparametrically

STA response

Solving for nonlinearity nonparametrically

STA response

Solving for nonlinearity nonparametrically

Projecting onto an axis orthogonal to the STA

Projecting onto an axis orthogonal to the STA

Projecting onto an axis orthogonal to the STA

was an accelerating nonlinearity typical of the great majority ofRGCs recorded. Some saturation was also evident at high valuesof the generator signal; this was commonly observed with highcontrast stimuli. Were RGC light responses linear in the presentconditions, the functions in Figure 3, B and D, would be linear.The departure from this prediction indicates that the nonlinearsecond stage of the LN model captured a significant feature of thelight response that can create systematic errors in a more restric-tive linear analysis of adaptation (see below).

Response nonlinearity in RGCs does not dependon contrastTo examine temporal contrast adaptation, RGC light responseswere characterized as above using random flicker stimuli of lowand high contrast. Thus, the stimulus that controlled the state ofadaptation was simultaneously used to probe light responses inthat state. Because the slowest known component of contrastadaptation operates over tens of seconds (Fig. 2) (Smirnakis etal., 1997; Kim and Rieke, 2001), responses from the first minuteof recording (more than three slow adaptation time constants)(Fig. 2) at each contrast level were excluded to confine analysis tothe steady state.

Figure 4A shows low and high contrast STAs from a represen-tative OFF cell. The corresponding nonlinearities are shown inFigure 4B. Because the units of the generator signal in the LNmodel are indeterminate, the linear filter in each condition isknown only up to a single, arbitrary scale factor. However, thelinear filters obtained with different contrasts can be meaning-

fully compared if scaling the amplitude of the high contrast linearfilter, and consequently scaling the abscissa of the high contrastnonlinearity, brings the nonlinearities for low and high contrastinto register, as is shown in Figure 4D (see Materials and Meth-ods for details). This superposition of nonlinearities in high andlow contrast implies that the changes in visual signaling caused bycontrast adaptation were attributable to changes in the linearfilter, shown in Figure 4C.

Figure 3. Characterization of light response in one ON cell (A, B) andone OFF cell (C, D) simultaneously recorded in salamander retina. A, C,The spike-triggered average L-cone contrast during random flicker stim-ulation is plotted as a function of time relative to the spike. This isproportional to the linear filtering of recent visual inputs. B, D, Spike rateis shown as a function of the estimated generator signal (stimulusweighted by linear filter), averaged over many time points during stimu-lation. Vertical (horizontal) error bars indicate the SE of spike rate(generator signal) for each such average; most error bars are smaller thanthe symbols. Smooth curve is a parametrized form of the cumulativenormal distribution, shifted and scaled to fit the data. Stimulus: 33 Hzbinary random flicker, 34% L-cone contrast.

Figure 4. Effect of contrast adaptation on light responses in salamanderRGCs. A, STAs for a single OFF cell obtained with low (17%, blacktrace) and high (34%, gray trace) contrast stimulation. Stimulus: 33 HzL-cone binary random flicker. B, Corresponding nonlinearities for lowcontrast (E) and high contrast (!). Error bars represent !1 SEM (see Fig.3). C, Linear filters: the low contrast STA, and the high contrast STAscaled by 0.35, are shown with black and gray lines, respectively. The highcontrast filter obtained with linear analysis is shown with a dashed grayline. For comparison with the low contrast filter, this was scaled so that itspeak divided by the peak of the low contrast filter equals the ratio of thepeaks of the high and low contrast filters obtained with linear analysis. D,Superimposed nonlinearities from four repeats of low contrast stimula-tion, and three repeats of high contrast stimulation with abscissa scaled by0.35. E, Peak sensitivity (solid black) and time to zero crossing (dashedgray) of the linear filter relative to the first low contrast filter for alter-nating low and high contrast stimulation. F, Fractional change in peaksensitivity and time to zero (relative to low contrast) for 24 simulta-neously recorded cells including the cell in A–E.

9908 J. Neurosci., December 15, 2001, 21(24):9904–9916 Chander and Chichilnisky • Contrast Adaptation in Primate and Salamander Retina

input strength

- Chander & Chichilnisky 01

!"#

$%

!"#$%&&

&'( '()##*+#,-.

/0#)#*+#,-.

- Pillow etal, 2004time (s) 10

T= 50 msT=100 msT=150 msT= 200 msXXTTYSpace, X [deg]Time, T [msec]0 40300

T= 50 msT=100 msT=150 msT= 200 msXXTTYSpace, X [deg]Time, T [msec]0 40300

T= 50 ms

T=100 ms

T=150 ms

T= 200 ms

X

X

T

T

Y

Space, X [deg]

Tim

e, T

[m

sec]

0 4 0

300

- Ozhawa, etal

V1 simple cell

0.1 1 21

0

5

10

15

20

25

0.1 1 10 201 10

0

5

10

15

20

25

0.1 1 21

0

5

10

15

20

25

0.1 1 10 201 10

0

5

10

15

20

25

0.1 1 21

0

5

10

15

20

0.1 1 10 201 10

0

5

10

15

SF [cyc/deg] TF [Hz]

Response [s

pik

es/s

ec]

X [deg]

T [

ms]

0 10

0

400

0

700

0

300

0 3.5

0 5

A

B

C

LNP summary• LNP is the defacto standard descriptive model,

and is implicit in much of the experimental literature

• Accounts for basic RF properties• Accounts for basic spiking properties (rate code)• Easily fit to data• Easily interpreted• BUT, non-mechanistic, and exhibits striking

failures (esp. beyond early sensory/motor) ...

STA estimation errors

• Convergence rate[Paninski, ’03]

• Non-spherical stimuli can cause biases

• Model failures:

- symmetric nonlinearity (causes no change in STE mean)

- response not captured by 1D projection

- non-Poisson spiking behaviors

e(k̂) ∼σ

E("k · "s)·

√

d

N

σ = stim s.d., d = stim dim., N = nspikes

Number spikes / stimulus dimensions

Erro

r0 10 20 30 40

0.5

1

1.5

2

eigenvalue number

varia

nce

0 10 20 30 40

0.5

1

1.5

2

eigenvalue number

varia

nce

0 10 20 30 40

0.5

1

1.5

2

eigenvalue number

varia

nce

100

101

102

103

0

0.02

0.04

0.06

0.08

0.1

Bootstrap errorAsymptotic error

Example 1: “sparse” noise

STA

true

Example 2: uniform noise

true

• Symmetric nonlinearities and/or multi-dimensional front-end (e.g., V1 complex cells)

LNP limitations

• Symmetric nonlinearities and/or multi-dimensional front-end (e.g., V1 complex cells)

LNP limitations

➡ Subspace LNP

Classic V1 models

Simple cell

Complex cell +

!" !##

$ %$ &$'

%

(

$ %$ &$

$)*

$)+

'

,-./0,-12./345,-126,5789:2

!" !##

$ %$ &$

$);

'

$ %$ &$

'

');

<%$$ <'$$ $

$ %$$

'

&$<%$$ <'$$ $

4/:7<:4,=86,5789

:7,0353:26,5789

8>?04582!##2.855

7,082@0:A

8>?04582!"2.855

!

1?,-

1?,-

$ %$ &$

'

"

#

$ % &

B

-839/-2%

:7,0353:26,5789

4/:7<:4,=826,5789

-/-5,-8?9,7C

B

:7/.D?:7,.:4,=,-1

./345,-16,5789:

-839/-2'

'

( !##20/:?,.!"20/:?,.

E/3458F2G4,=,-12H/F85

8>

Figure 1: Multicell encoding model and parameter fits. a, Model schematic for two coupled neu-rons: each cell has a stimulus filter (receptive field), post-spike filter, and “coupling filters” thatcapture dependence on recent spikes in other neurons. Filter outputs are summed, passed throughan exponential nonlinearity, and generate spikes via an instantaneous (Poisson) spike generator. b,Mosaics of 11 ON and 16 OFF retinal ganglion cell receptive fields (RFs), tiling a small region ofvisual space. Ellipses represent 1 SD of a Gaussian fit to each RF center, and gray lines indicate anunderlying 4x4 lattice of stimulus pixels. (c-d), Paramaters fit to an example ON cell. c, Temporaland spatial components of center (red) and surround (blue) filter components; the outer product ofthese components gives center and surround spatiotemporal filters, whose difference is the full stim-ulus filter. d, Exponentiated post-spike filter, which effectively multiplies the spike rate following aspike at time zero. (Relative spike rate drops to zero following a spike and recovers with a slightovershoot before settling to 1). e, Connectivity diagram and coupling filters (below) from other cellsin the network. Black filled ellipse is the cell’s RF center, and blue and red lines show connectionsfrom neighboring OFF and ON cells (with coupling strength indicated by line thickness). Below,exponentiated coupling filters show the multiplicative effect of a neighboring spike on instantaneousspike rate. (f-h), Analagous plots for parameters of example OFF cell.

spiking history of other cells (Figure 1a). It is this functional coupling that allows the model tocapture response correlations above and beyond those induced by the stimulus, as it stochasticspiking in one cell is able to influence the spiking activity in other cells on multiple timescales.

We fit the model to a population of 27 parasol retinal ganglion cells (RGCs), stimulated with120-Hz spatiotemporal binary white noise (spatial flicker), recorded in vitro from a small patch ofmacaque retina. The cells’ receptive fields (RFs), which can be divided into ON and OFF classesbased on the sign of their light responses, form two complete mosaics over a small region of visualspace (fig. 1b). The full model consists of stimulus and spike-history filters for each cell, plusa total of 27 · 26 = 702 coupling filters (2 for each pair of cells), fit using maximum likelihood,which is tractable due the model’s simple and well-behaved (i.e. log-concave) likelihood function[8]. Using a likelihood-based “pruning” procedure, we were able to reduce the number of couplingfilters to a minimal sufficient set, which reduces the model’s computational complexity and providesan estimate of the network’s functional connectivity [7] (see Methods).

3

!" !##

$ %$ &$'

%

(

$ %$ &$

$)*

$)+

'

,-./0,-12./345,-126,5789:2

!" !##

$ %$ &$

$);

'

$ %$ &$

'

');

<%$$ <'$$ $

$ %$$

'

&$<%$$ <'$$ $

4/:7<:4,=86,5789

:7,0353:26,5789

8>?04582!##2.855

7,082@0:A

8>?04582!"2.855

!

1?,-

1?,-

$ %$ &$

'

"

#

$ % &

B

-839/-2%

:7,0353:26,5789

4/:7<:4,=826,5789

-/-5,-8?9,7C

B

:7/.D?:7,.:4,=,-1

./345,-16,5789:

-839/-2'

'

( !##20/:?,.!"20/:?,.

E/3458F2G4,=,-12H/F85

8>

Figure 1: Multicell encoding model and parameter fits. a, Model schematic for two coupled neu-rons: each cell has a stimulus filter (receptive field), post-spike filter, and “coupling filters” thatcapture dependence on recent spikes in other neurons. Filter outputs are summed, passed throughan exponential nonlinearity, and generate spikes via an instantaneous (Poisson) spike generator. b,Mosaics of 11 ON and 16 OFF retinal ganglion cell receptive fields (RFs), tiling a small region ofvisual space. Ellipses represent 1 SD of a Gaussian fit to each RF center, and gray lines indicate anunderlying 4x4 lattice of stimulus pixels. (c-d), Paramaters fit to an example ON cell. c, Temporaland spatial components of center (red) and surround (blue) filter components; the outer product ofthese components gives center and surround spatiotemporal filters, whose difference is the full stim-ulus filter. d, Exponentiated post-spike filter, which effectively multiplies the spike rate following aspike at time zero. (Relative spike rate drops to zero following a spike and recovers with a slightovershoot before settling to 1). e, Connectivity diagram and coupling filters (below) from other cellsin the network. Black filled ellipse is the cell’s RF center, and blue and red lines show connectionsfrom neighboring OFF and ON cells (with coupling strength indicated by line thickness). Below,exponentiated coupling filters show the multiplicative effect of a neighboring spike on instantaneousspike rate. (f-h), Analagous plots for parameters of example OFF cell.

spiking history of other cells (Figure 1a). It is this functional coupling that allows the model tocapture response correlations above and beyond those induced by the stimulus, as it stochasticspiking in one cell is able to influence the spiking activity in other cells on multiple timescales.

We fit the model to a population of 27 parasol retinal ganglion cells (RGCs), stimulated with120-Hz spatiotemporal binary white noise (spatial flicker), recorded in vitro from a small patch ofmacaque retina. The cells’ receptive fields (RFs), which can be divided into ON and OFF classesbased on the sign of their light responses, form two complete mosaics over a small region of visualspace (fig. 1b). The full model consists of stimulus and spike-history filters for each cell, plusa total of 27 · 26 = 702 coupling filters (2 for each pair of cells), fit using maximum likelihood,which is tractable due the model’s simple and well-behaved (i.e. log-concave) likelihood function[8]. Using a likelihood-based “pruning” procedure, we were able to reduce the number of couplingfilters to a minimal sufficient set, which reduces the model’s computational complexity and providesan estimate of the network’s functional connectivity [7] (see Methods).

3

V1 simple cell

space sf

eigenvalue number0 50 100 150 200 250

0.2

0.6

1

1.6

1.8

STA

varia

nce

time

tf

STC

[Rust, Schwartz, Movshon, Simoncelli, ‘05]

• Symmetric nonlinearities and/or multi-dimensional front-end (e.g., V1 complex cells)

• Linear front end insufficient for non-peripheral neuraons

LNP limitations

➡ Subspace LNP

➡ Cascades / fixed front-end nonlinearities

• Symmetric nonlinearities and/or multi-dimensional front-end (e.g., V1 complex cells)

• Linear front end insufficient for non-peripheral neuraons

• Responses depend on spike history, other cells

LNP limitations

➡ Subspace LNP

➡ Cascades / fixed front-end nonlinearities

➡ Recursive models (GLM)

stimulus filter pointnonlinearity

probabilisticspiking

Linear-Nonlinear-Poisson (LNP)

[Truccolo et al ‘05;Pillow et al ‘05]

stimulus filter

post-spike

exponentialnonlinearity

+

probabilisticspiking

waveform

Recursive LNP

modelparameters

stimulus & spike train

• Critical: Likelihood function has no local maxima [Paninski 04]

maximize likelihood

modelparameters

stimulus & spike train

• Critical: Likelihood function has no local maxima [Paninski 04]

28

!

"!!

#!!

$%&'()*+,*-

(

(

! !."/ !./ !.0/ 1

!

/

2%$3%45'

&36'()*-

!."/7 !.8

9:;

<=>

:<?

:<?

9:;

<=>

&36'()*-

! !."/ !./ !.0/ 1

&36'()*-

@ A

;

B

Figure 2: Example predictions of a single retinal ganglion ON-cell’s activity using the GLencoding model with and without spike history terms. Conventions are as in Figs. 3 and 4in (Pillow et al., 2005b); physiological recording details as in (Uzzell and Chichilnisky, 2004;Pillow et al., 2005b). A: Recorded responses to repeated full-field light stimulus (top) of trueON-cell (“RGC”), simulated LNP model (no spike history terms; “LNP”), and GL modelincluding spike-history terms (“GLM”). Each row corresponds to the neuron’s spike trainresponse during a single stimulus presentation. Peristimulus rate and variance histograms areshown in panels C and D, respectively. B: Magnified sections of rasters, with rows sorted inorder of first spike time within the window in order to show spike timing details. Note thatthe predictions of the model including spike history terms are in each case more accurate thanthose of the Poisson (LNP) model.

Here the term b(t) encodes the dependence of the firing rate as a function of time since thebeginning of the trial; the spike history terms h(.) play the same role as above. We may fitb(.) and h(.) simultaneously, due to the joint concavity of the loglikelihood in the parameters(b(.), h(.)).

0.7 Multiplicative models of spike history-dependence; connections to re-newal models

In the next two sections, we will point out some connections of the above likelihood-basedtechniques to related models that have appeared in the literature. The first such modelwe will consider was developed in order to correct the main deficiency of the basic LNPmodel, namely its Poisson nature. We introduced an “additive” model for incorporatinghistory dependence above (that is, the spike history effects entered in an additive manner,through the terms

∑

j hjnt−j); a simple alternate way to model the spike-history effects (e.g.,

9

rLNP model

stimulus

stimulus filter

post-spike

exponentialnonlinearity

+

probabilisticspiking

waveform

rLNP model

stimulus

stimulus filter

post-spike

exponentialnonlinearity

+

probabilisticspiking

waveform

+

rLNP model

stimulus

stimulus filter

post-spike

exponentialnonlinearity

+

probabilisticspiking

waveform

+

couplingwaveforms

Multi-cell rLNP, with spike coupling

30

Equivalent model diagram

x(t)

y1(t)

yn(t)

K

h11

h1n

Equivalent model diagram

30

Equivalent model diagram

x(t)

y1(t)

yn(t)

K

h11

h1n

Equivalent model diagram

modelparameters

stimulus & spike trains

maximize likelihood

modelparameters

stimulus & spike trains

maximize likelihood

modelparameters

stimulus & spike trains

• Again: Likelihood function has no local maxima [Paninski 04]

Methods

spatiotemporal binary white noise (8 minutes at 120Hz)

macaque retinal ganglion cell (RGC) spike responses

Methods

(ON and OFF parasol)

Text

[Pillow, Shlens, Paninski, Chichilnisky, Simoncelli - unpublished]

!

"#$%&"'(

)*+,$-$)'.+-*#%

/&)*0)/+1#'.+-*#%

#2/&"#"*+3-"&"-+"#3%+*4

!

/%&535+-+)*+6)/+1+"7

6&$/-+"7.+-*#%)

"#$%&"'8

9

:

; (; <;8

(

=

; (; <;

;>?

;>@

8

; (; <;;

8

/&)*0)/+1#.+-*#%

)*+,$-$)'.+-*#%

+"6&,+"7'6&$/-+"7'.+-*#%)'

#23,/-#'ABB'6#--'/3%3,#*#%)

*+,#'C,)D

#23,/-#'AE'6#--'/3%3,#*#%)

ABB',&)3+6AE',&)3+6

F&$/-#G'H/+1+"7'I&G#-

0(;; 08;; ;

; (; <;

8

8>J

; (; <;

;>J

8

; (; <;;

8

0(;; 08;; ;

K

F

73

+"7

3+"

AE

AE ABB

ABB

Figure 1: Model schematic and parameter fits. (A) Model schematic showing parameters for twocoupled neurons. Each cell has a stimulus filter (receptive field), post-spike filter, and “couplingfilters” that capture dependence on spikes in other neurons. Filter outputs are summed and passedthrough an exponential nonlinearity to yield spike rate for an instantaneous (Poisson) spike generator.(B) Mosaics of 11 ON and 16 OFF retinal ganglion cells tiling a single region of visual space. Ellipsesshow 1 SD of a Gaussian fit to each receptive field (RF) center, and gray lines indicate a common4x4 lattice of stimulus pixels. (C) Paramaters fit to an example ON cell. Left: temporal and spatialcomponents of center (red) and surround (blue) filter components, whose difference is the full stimulusfilter. Middle: exponentiated post-spike filter, which multiplies the spike rate following a spike attime zero. (Relative spike rate drops to zero following a spike and recovers with a slight overshootbefore settling to 1). Right: coupling filters from other cells in the network. Above, black filledellipse is the cell’s RF center, and blue and red lines show connections from neighboring OFF andON cells (RFs indicated by gray ellipses; line thickness indicates strength of the coupling). Below,exponentiated coupling filters show the multiplicative effect of a spike in a neighboring cell on spikerate. (D) Analagous plots for parameters fit to an example OFF cell.

08 ; 8

AE0AEABB0ABBAE0ABB

08

;

8/3+%L+)#'MB'6&%%'C%(D

F

6#"*#%)

')$%%&$"G)

;>J 8 8>J

;>J

8

8>J

.$--'

,&

G#

-'

MB')+N#'C)*G#OD

$"6&$/-#G',&G#-

:

$"

6&

$/

-#G

.$--'

,&

G#

-

AE)$%%&$"G)

ABB)$%%&$"G)

9

Figure 2: Changes in receptive field estimates induced by coupling. (A) Ellipses show 1SD ofGaussian fit to RF surround of each cell under full model (above) and “uncoupled” model (below),showing that coupling induces an effective decrease in surround size. (B) Scatter plot of center andsurround RF sizes under full and uncoupled model. (C) Correlation coefficient between all pairs ofspatiotemporal stimulus filters, indicating that coupling decorrelates stimulus filters.

Comparison to an uncoupled model

In order to understand the effects of functional coupling in the recurrent-LNP model on the pop-ulation response, we can compare the performance of the full model with that of a reduced model5

Example ON cell

Example OFF cell

[Pillow, Shlens, Paninski, Chichilnisky, Simoncelli - unpublished]

20

30

20

30

20

30

20

30

-50 0 50

20

30

time (ms)

20

30

-50 0 50time (ms)

Cross-Correlations

ONcells

OFFcells

GLMRGC

rate

(sp/

s)

rLNP

Cross-Correlations

20

30

20

30

20

30

20

30

-50 0 50

20

30

time (ms)

20

30

-50 0 50time (ms)

ONcells

OFFcells

GLM (no coupling)RGC

rate

(sp/

s)

rLNP

!!"##$%&'(# ")*&"+#(',-.!"##$%&'(# ")*&"+#(',-.

τ/$0%12 τ3

456($01+712

"

*(##$)"%8(4

/3/9/:/;/</=/>/?3@3/33393:3;3<3=

/9 /: /; /< /= /> /? 3@ 3/ 33 39 3: 3; 3< 3=

/ 3 9 : ; < = > ? /@

39

:;

<=

>?

/@//

/ 3 9 : ; < = > ? /@ //

ABCADD$ABCAB$ ADDCADD$

;@$%1

=;$1+71

9@$1+71#

$

%&

/

3

9

:

;

< =

>

?

/@

//

/3

/9

/:/;

/<

/= />

/?

3@3/

33

39

3:

3;

3<

3=

,-. !"##$%&'(# ")*&"+#('$%&'(#

'

@ 9@@@

9@@

+(5E$01+712

%&'(#

,-.

!"##

")*+#F

AB$

ADD$

Figure 2: Analysis of response correlations. a, RF mosaics with arbitrary numbering of cells. bCross-correlations functions (CCFs) bewteen all ON cell pairs, calculated on RGC responses (blackdotted) and simulated responses of the full (red) and uncoupled models (blue); i, jth plot shows theCCF between cell ith and cell j. Baseline has been subtracted off so that all graphs have the sameaxes, in units of spikes/s above (or below) the first cell’s mean spike rate. Gray region highlights theset of CCFs associated with cell 5 (fig.1,c-e). c Cross-correlations between all OFF cell pairs, withgray region highlighting the correlations of cell 15 (fig.1,f-h). d CCFs for all ON-OFF pairs, withvertical axis rescaled to span only 30 sp/s. e, Third-order (triplet) CCF between three adjacent ONcells, showing the instantaneous spike rate of cell 5 as a function of the relative spike time in cells4 and 8, for spike responses of the RGCs (left), full model (middle) and uncoupled model (right).f Analogous triplet CCF for OFF cells 15, 16 and 22. g Comparison of triplet CCF peak heightin RGC responses and model simulations, for randomly selected triplets of adjacent ON (open) andOFF cells (filled symbols).

How do correlations affect encoding?

Once we have verified that the model captures the essential correlation structure in the populationresponse, we can turn to ask how correlations affect our understanding of the encoding process.A bit surprisingly, the independent model predicts single-cell responses to repeated stimuli withroughly the same fidelity as the full model (i.e., capturing 80-95% of the PSTH variance in 26of 27 cells, fig. 3a-c). However, if we compute the log-likelihood of novel spike trains under both

5

Single-trial spike predictionRGC responses single-trial predictions (neighbor spikes “frozen”)

Single-trial spike predictionRGC responses single-trial predictions (neighbor spikes “frozen”)

Equivalent model diagram

x(t)

y1(t)

yn(t)

K

h11

h1n

Single-trial spike predictionRGC responses single-trial predictions (neighbor spikes “frozen”)

Equivalent model diagram

x(t)

y1(t)

yn(t)

K

h11

h1n

1 1.5 2

1

1.5

2

spike prediction (bits/sp)

LNP60 80 100

50

60

70

80

90

100PSTH prediction (%var)

LNP

full

mod

el

RGC responses single-trial predictions (neighbor spikes “frozen”)

Single-trial spike prediction

1 1.5 2

1

1.5

2

spike prediction (bits/sp)

LNP60 80 100

50

60

70

80

90

100PSTH prediction (%var)

LNP

full

mod

el

RGC responses single-trial predictions (neighbor spikes “frozen”)

Single-trial spike prediction

!"#$%&'()"*%+,)&-".("/#!

01+2!

1

311

("2&

4

)*(&

+5!

,6!

7

)$.+)*!(&)

89+.&%%8::+.&%%

;1 <1 =11

;1

<1

=11

,!(>+,)&-".("/#5?@*)7

A#./A,%&-+2/-&%

BA%%+

2/

-&

%

=C0 3

=C0

3

!,"D&+,)&-".("/#+%/$'%"+5E"(!6!,7

F

)$.

A#./A,%&-

2/-&%

BA%%

2/-&%

1C= 1C01C=

1C0

./))+./&B+G"(>()A&+!,"D&!

()A&+,!(>

!"#

$%&

'()"

*%

)*(&

+,)&

-".

("/

#

H

I J

1 1C0 =1

=11

311

("2&+5!7

)*(&

+5!

,6!

7

Figure 4: Model validation and comparison at predicting responses to novel stimuli. (A) Rastershowing responses of an ON RGC to 25 repeats of a 1-s stimulus, and simulated responses of uncoupled(blue) and full models (red) to the same stimulus. Below, peri-stimulus time histogram (PSTH) ofthe RGC, uncoupled and coupled model; both account for ≈ 84% of the variance of the true PSTH.(B) Scatter plots of PSTH prediction and spike-train log-likelihood under full and uncoupled models.Curiously, coupling does not seem to improve mean rate prediction, but leads to a ≈ 20% improvementin predicting population spike trains. (C) Magnified 150-ms portion of RGC raster (indicated bygray box in A). Red dots show RGC spike times during individual trials for which rate predicitonsare plotted in D. (D) Illustration of single-trial prediction using coupled model (8 examples). Eachplot shows red dots (top) indicating true spike times of RGC during a single trial. Raster (below)shows 50 sample spike trains generated under the (model-based) probability distribution given byassuming spike responses of all other cells fixed for this trial. Red trace (below) shows single-trial rateprediction (average of model-generated spike trains), compared with true PSTH of the cell duringthis interval (black trace, identical in all plots). (E) Correlation of true spike trains with averageresponse (ordinate) and with model-based single-trial rate predictions (abcissa), indicating that themodel accounts for significant trial-to-trial variability in the neural response.

carried by trial-averaged mean rate, then perhaps not!). The probabilistic spiking model providesa natural framework for addressing this question. Assuming that the model provides an accu-rate model of multi-neuronal responses, we can perform Bayesian optimal decoding of populationresponses to stimuli. That is, we can find the best possible estimate of the stimulus under theinterpretation provided by the model. We can then compare the estimate given by the full modelwith that of the uncoupled model, which provides the best possible estimate under the assumptionthat neurons respond independently.

8

full

mod

el

uncoupled model

!"#$%&

'(!$")%'(*)+,!(-.-/

0!"*'%1$!--

2

32

42

52

62#700

7"890

!"

*"8$:!";&$:*0

<'=*-!'":*8$:!";

9+%>-/

9+-/

9+->%/

(*&9$%'0)#%*?7*"8=)+@A/

!"#$%&

'(!$

")+,!(-

/

2 42 62 B22

3

#

Figure 4: Comparison of linear and Bayesian decoding of population spike responses. a, Schematic:Bayesian decoding of a stimulus chunk from a set of observed spike times. The stimulus prior p(s),multiplied by the model-defined likelihood p(r|s) defines the posterior p(s|r), whose mean is theBayes’ least-squares estimate. Gray boxes indicate the stimulus chunk to be decoded and the set ofspike times relevant to its decoding. a, Information rate under various decoding methods. 150-mssingle-pixel stimulus segments (18 time bins) were decoded from population spike responses usinglinear decoding and Bayesian decoding of the Poisson, uncoupled and full models. The full modelpreserves 21% more information than the uncoupled model, indicating the importance of correlationsto the information content of the population response. c, Mutual information decomposed as afunction of temporal frequency for various decoding methods.

uncoupled models exhibit no difference in predicting the mean response to repeated stimuli (i.e.,capturing 80-95% of the PSTH variance in 26 of 27 cells, fig. 3a-c). However, the log-likelihood ofnovel single-trial spike trains exhibits a average 13% increase in information under the full model(fig. 3d), indicating that preserving correlations leads to significantly better prediction of novelspike responses. How can we make sense of this apparent discrepancy?

Consider how the population response constrains the response of an individual cell on a singletrial. Even though the coupled and uncoupled models yield similarly accurate predictions of thecell’s average response, on any single trial, the spike trains of other cells in the population carryinformation about the response on that trial, since the coupling filters determine when spikes aremost likely to occur relative to the spikes in other cells. To flesh out this intuition, used the modelto make single-trial predictions of a cell’s response using both the stimulus and the spiking activityin the remainder of the population (fig. 3e-f). For each trial, we sampled spikes for a single cellusing the model-defined probability distribution over those spikes times (see Methods), effectivelycreating a response raster where both the stimulus and population activity were presented manytimes. Averaging this raster gives a prediction of the cell’s spike rate for that specific trial. Wecompared single-trial rate predictions with the cell’s true spike times on many different trials, andfound that, on nearly every trial, the model-based prediction correlates better with the observedspike times than the average response (i.e./ PSTH) of the cell itself (fig. 3g). Thus, correlationscan be exploited to predict spike times more precisely than if we had access to the (true) averageresponse to a novel stimulus, and much of the trial-to-trial variability in a cell’s response can beattributed to population activity.

Although the model provides a more accurate description of multi-neuronal encoding, we have notyet assessed the importance of correlations to the stimulus-related information content of populationspike trains. If the stimulus information can be read out of the time-varying spike rates, withnoise assumed to be independent across neurons, then improved spike train prediction afforded by

6

LNP rLNP crLNPopt.linear

Stim

ulus

info

rmat

ion

(bits

/s)

Decoding