Prapun B-Exam

60
Introduction Sources of Variability and Threshold Derivation Information-Theoretic Analysis of IF Neuron Conclusion Capacity Analysis of Neurons with Descending Action Potential Thresholds Prapun Suksompong Electrical and Computer Engineering Cornell University, Ithaca, NY 14853 [email protected] Final Examination for the Doctoral Degree (“B” Exam) July 24, 2008 Prapun Suksompong Capacity Analysis of Neurons

Transcript of Prapun B-Exam

Page 1: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Capacity Analysis of Neurons with DescendingAction Potential Thresholds

Prapun Suksompong

Electrical and Computer EngineeringCornell University, Ithaca, NY 14853

[email protected]

Final Examination for the Doctoral Degree (“B” Exam)July 24, 2008

Prapun Suksompong Capacity Analysis of Neurons

Page 2: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Outline

Introduction

Sources of Variability for the ISIs and Derivation of the Threshold

Information-Theoretic Analysis of IF Neuron

Conclusion

Prapun Suksompong Capacity Analysis of Neurons

Page 3: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

IntroductionNeuron MorphologyIntegrate-and-Fire NeuronsGoal

Sources of Variability for the ISIs and Derivation of the Threshold

Information-Theoretic Analysis of IF Neuron

Conclusion

Prapun Suksompong Capacity Analysis of Neurons

Page 4: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Neuron Morphology

A neuron is the basic working unit of the nervous system.

A typical neuron has threefunctionally distinct parts, called

I dendrites,

I soma, and

I axon.

The junction between twoneurons is called a synapse.

Nucleus

Axon

Dendrite

Axon Terminals

Myelin Sheath

Cell Body (Soma)

Presynaptic Axom

Terminal Synaptic Cleft

Postsynaptic Dendrite

Ion Channel

Synapse

Axon from another neuron

Node of Ranvier

Synaptic Vesicle

Axon Hillock

Prapun Suksompong Capacity Analysis of Neurons

Page 5: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Action Potentials (Spikes)

Looking at a synapse, we refer to the sending neuron as thepresynaptic neuron and to the receiving neuron as thepostsynaptic neuron.

postsynaptic presynaptic

axon synapse

The neuronal signals consist of short electrical pulses called actionpotentials (APs) or spikes. A chain of APs emitted by a singleneuron is called a spike train.

Prapun Suksompong Capacity Analysis of Neurons

Page 6: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Quantal Synaptic Failure (QSF)

postsynaptic presynaptic

axon synapse

I Synaptic failure: It is possible that an AP fails to get“across” the synapse.

I We may model a synapse as a Z -channel.

I Spikes which successfully cross the synapse then propagatedown to soma.

Prapun Suksompong Capacity Analysis of Neurons

Page 7: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Integrate-and-Fire Neurons

I Assumption: ∼ 104

pre-synaptic neurons.

I True in cortex (higher brainfunctions).

Prapun Suksompong Capacity Analysis of Neurons

Page 8: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Integrate-and-Fire Neurons

I Spikes generated when themembrane potentials hit thethresholds.

I Descending thresholds.

Thank you for coming to this talk. This work is a joint effort between Prof. Berger and me. We also got quite some help from Dr. Levy over here. For this session, we will focus on the timing jitter which occurs in communication between neuron. First, let’s recall that a neuron receives spike trains from many neurons and for us we will assume that the number of incoming connections is large, say, on the order of ten-thousand. The neuron in the middle of the figure sums up the contribution from each incoming spikes and when its membrane potential reaches a specific value which everyone call the “threshold”, we get a spike which propagate to another neuron. Now, instead of a constant threshold whose value is fixed at a specific level, the thresholds which are drawn here are in fact decreasing. We will return to this later.

Descending Threshold

Ascending Membrane Potential

Spike Train time

time

• First jitter: Spike generation • Poisson approximation

Prapun Suksompong Capacity Analysis of Neurons

Page 9: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

(Leaky) Integrate-and-Fire Model: LIF or IF

Let τ1, τ2, τ3, . . . be the sequence of time that the spikes arrive atthe spike generating region. The membrane potential at time t isthen

X (t) =∑m

h (t, τm,Ym) =∑m

Ymh(t − τm).

This is the “integrate” part of the integrate-and-fire neuron.

I Ym is the weight for the mth

spike due to propagationloss, synaptic strength,synaptic failure, etc.

I h is the shape function.

( )X t

( )2 2i iY h t τ+ +−

( )1 1i iY h t

( )i iY h t −τ

1iτ + 2iτ +

τ+ +−

i

time τ

Prapun Suksompong Capacity Analysis of Neurons

Page 10: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Integrate-and-Fire Neurons (Con’t)

I Constant bombardment of spikesleads to increase in membranepotential.

I As soon as the membrane potentialreaches a critical value orthreshold, the neuron “fires” anaction potential. Then, everythingresets.

I Refractory period: The time aftera AP is produced, during which itis impossible to generate anotherAP.

I Set T (t) to be ∞ during thisperiod.

 

time (t)

time (t) 

Threshold

Prapun Suksompong Capacity Analysis of Neurons

Page 11: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Integrate-and-Fire Neurons (Summary)

I ∼ 104 pre-synaptic neurons.

Thank you for coming to this talk. This work is a joint effort between Prof. Berger and me. We also got quite some help from Dr. Levy over here. For this session, we will focus on the timing jitter which occurs in communication between neuron. First, let’s recall that a neuron receives spike trains from many neurons and for us we will assume that the number of incoming connections is large, say, on the order of ten-thousand. The neuron in the middle of the figure sums up the contribution from each incoming spikes and when its membrane potential reaches a specific value which everyone call the “threshold”, we get a spike which propagate to another neuron. Now, instead of a constant threshold whose value is fixed at a specific level, the thresholds which are drawn here are in fact decreasing. We will return to this later.

Descending Threshold

Ascending Membrane Potential

Spike Train time

time

• First jitter: Spike generation • Poisson approximation

I Descending thresholds.

Prapun Suksompong Capacity Analysis of Neurons

Page 12: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Theoretical Approaches to Neuroscience

I We use IF model, but more biologically-realistic models exist(e.g. Hodgkin and Huxley [’52] model).

1. Too many parameters.I Physical measurements “fundamentally disturb cell properties”

2. Provide less insight.

I Biological structures have evolved via natural selection tooperate optimally.

I See the book Optima for Animals by R. McNeill Alexander.I What is the best strength for a bone?I At what speed should humans change from walking to

running?

Prapun Suksompong Capacity Analysis of Neurons

Page 13: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Information-Theoretic Optimization

Application of information theory has already found success inmany areas of neuroscience.

I Barlow’s “economy of impulses”[’59, ’69]I Minimize redundancy.

I Linsker’s InfoMax principle [’88, ’89]I Maximize the mutual information.

I Levy and Baxter’s energy-efficient coding [’96, ’02]I Maximize mutual information per unit energy expended.

Prapun Suksompong Capacity Analysis of Neurons

Page 14: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Neuron MorphologyIntegrate-and-Fire NeuronsGoal

Motivation and Goal

I Integrate-and-Fire (IF) model is very popular.

I The threshold function is a crucial element of the IF model.

I Little amount of work exists on deriving the form of thethreshold curve.

I In fact, using constant thresholding is also popular.I This leads to large jitter in the spike timing and hence

discourages the use of time coding.

Goal: Find (1) an expression for threshold curve under biologicallyrealistic constraints and (2) the optimal operating point of neuronunder such threshold.

Prapun Suksompong Capacity Analysis of Neurons

Page 15: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Introduction

Sources of Variability for the ISIs and Derivation of the ThresholdFirst Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Information-Theoretic Analysis of IF Neuron

Conclusion

Prapun Suksompong Capacity Analysis of Neurons

Page 16: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Three sources of variability for inter-spike intervals

1. Spike generation

2. Spike propagation

3. Time-of-arrival estimation

Prapun Suksompong Capacity Analysis of Neurons

Page 17: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

First jitter: Spike generationformula that governs how the membrane potential of this middle neuron rises given the combined incoming rate λ .

21 3λ λλλ= + + +

Now, of course, there is some jitter in the timing of the spike and that’s the focus of our paper today. The first source of jitter comes from the fact that the incoming spike trains on this side has in fact some jitter in them. Now, you may recall that we assume that number of these presynaptic neurons are large. That assumption allows us to say that the combined effect .. the superposed spike trains … is close to a Poisson process even though the individual spike train coming out of a single neuron is not a Poisson process. That allows us to find a tractable formula that governs how the membrane potential of this middle neuron rises given the combined incoming rate λ .

I Large number of presynaptic neurons allows Poissonapproximation for the superposed process.

I The membrane potential is governed by a filtered Poissonprocess.

Prapun Suksompong Capacity Analysis of Neurons

Page 18: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Approximation for the first timing jitter

I For fixed λ, different realizations of the membrane potentialcorrespond to different spiking times.

Membrane potential ( )X t

Threshold ( )T t

timeσ Time

I Filtered Poissonapproximation foramount of variationin vertical direction[Parzen’62].

I Linear approximationfor amount ofvariation in horizontaldirection.

Prapun Suksompong Capacity Analysis of Neurons

Page 19: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Approximation for the first timing jitter (con’t)

σtime (τ) ≈ H (τ)

T (τ) h (τ)− T ′ (τ) H (τ)

√T (τ) c2H2 (τ)

c1H (τ)

Membrane potential ( )X t

Threshold ( )T t

timeσ Time

I h : shape function.I e.g. exponential

Membrane potential ( )X t

Threshold ( )T t

timeσ Time

The figure here shows different realizations of the membrane potentials for a fixed combined incoming rate λ . Here, we see that the randomness from the Poisson arrivals causes fluctuation in the time that the membrane potentials hit the threshold. Under some linear approximation, we can relate the jitter in the vertical direction to the one in horizontal direction. This then gives us the formula for the magnitude of the timing jitter as a function of the spike time τ .

Here, h is the shape function which describes how the membrane potential changes in response to a single input spike. For the usual leaky integrate-and-fire model, this h starts with some amplitude and then decay exponentially.

( )h t

For conciseness, we define these two integrations which get used in the formula here.

I H (t) =

t∫0

h (µ)dµ.

I H2 (t) =t∫

0

h2 (µ)dµ.

Prapun Suksompong Capacity Analysis of Neurons

Page 20: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Approximation for the first timing jitter (con’t)

σtime (τ) ≈ H (τ)

T (τ) h (τ)− T ′ (τ) H (τ)

√T (τ) c2H2 (τ)

c1H (τ)

Membrane potential ( )X t

Threshold ( )T t

timeσ Time

I c1 and c2 are constantswhich depend on thedistribution of the weight(Ym) for each spike.

Recall:X (t) =

∑m Ymh(t − τm).

Prapun Suksompong Capacity Analysis of Neurons

Page 21: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Jitter in rate estimation

I The only information contained in a Poisson process is its rateλ.

I Different λ’s ⇒ different spiking times τ ’s.

PSP for Large λ

PSP for Small λ

Descending Threshold

Time

However, this spike time has some jitter, so the $\lambda$ estimation also have some error. We then go on and approximate this error:

We also consider two other sources of jitters which I won’t go into details. The second jitter is the randomness in the length of time a spike takes to propagate to a synapse on a particular neuron. IT is on the order of 10 microseconds. The third jitter is the time-of-arrival estimation error.

I Spike times vary inversely with λ.

λ(τ) ≈ T (τ)

c1H (τ)

I Error in rate estimation:

σλ (τ) ≈ 1

c1H (τ)

√T (τ) c2H2 (τ)

c1H (τ).

Prapun Suksompong Capacity Analysis of Neurons

Page 22: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Second and Third Jitters

I Propagation Time.

I Time-of-Arrival EstimationError (radar rangingproblem):(

4π2 2Es

N0f 2

)−1

.

I

√f 2: Gabor-bandwidth.

I ES : Signal energy.I N0

2 : Spectral height of theNoise.

I Small: < 10µs.

For the together two sources of jitter. I won’t go into details. The second jitter is the randomness in the length of time a spike takes to propagate to a synapse on another neuron. It is on the order of 10 microseconds. The third jitter is the time-of-arrival estimation error; that is, if this neuron tries to measure the inter-spike interval, it needs to find out what time a spike arrives. We borrow some formula from the Radar guys shown here because this is exactly the problem that they call the radar ranging problem. The error depends on the signal-to-noise ratio which is shown here as this Es over N0 here. It also depends on the bandwidth for the shape of the action potential. The amount of error here is about 10 microseconds as well.

Prapun Suksompong Capacity Analysis of Neurons

Page 23: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Deriving The Threshold Curves

Recall: Deviation in time:

σtime (τ) =H (τ)

T (τ) h (τ)− T ′ (τ) H (τ)

√T (τ) c2H2 (τ)

c1H (τ).

We consider the thresholds which

1) preserve timing jitter σtime (τ) ≡ σtime,0, or

2) preserve relative timing jitter σtime(τ)τ ≡ σ%time,0

across spiking times (or spiking frequencies) of interest.

Prapun Suksompong Capacity Analysis of Neurons

Page 24: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Deriving The Threshold Curves

Recall:

(a) Deviation in time: σtime (τ) = H(τ)T (τ)h(τ)−T ′(τ)H(τ)

√T (τ)c2H2(τ)

c1H(τ) .

(b) Deviation in λ estimation: σλ (τ) = 1c1H(τ)

√T (τ)c2H2(τ)

c1H(τ) .

We consider the thresholds which

1) preserve timing jitter σtime (τ) ≡ σtime,0, or

2) preserve relative timing jitter σtime(τ)τ ≡ σ%time,0, or

3) preserve jitter in λ estimation σλ (τ) ≡ σλ,0, or

4) preserve relative jitter in λ estimation σλ(τ)λ ≡ σ%λ,0, or

5) preserve jitter in lnλ estimation

across spiking times (or spiking frequencies) of interest.

Prapun Suksompong Capacity Analysis of Neurons

Page 25: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Deriving The Threshold Curves (con’t)

I Constant timing jitter level:

T ′ (t) = T (t)

(h (t)

H (t)

)−√

T (t)

(1

σtime,0

√c2H2 (t)

c1H (t)

).

I Constant relative-timing-jitter level:

T ′ (t) = T (t)

(h (t)

H (t)

)−√

T (t)

(1

tσ%time,0

√c2H2 (t)

c1H (t)

).

I Preserve relative error in λ estimation:

T (t) = c0H2 (t)

H (t).

Prapun Suksompong Capacity Analysis of Neurons

Page 26: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

The differential equations are Bernoulli equations of the form

T ′ (t) = T (t) P (t)−√

T (t)Q (t) .

They can be reduced to linear equation by introducingv(t) =

√T (t) which gives

v ′ (t) =1

2P (t) v (t)− 1

2Q (t) .

Linear! The solution is

v (t) = v (t0)φ (t, t0)−t∫

t0

1

2φ (t, τ) Q (τ)dτ,

where φ (t, s) = e

t∫s

12P(τ)dτ

.

Prapun Suksompong Capacity Analysis of Neurons

Page 27: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Comparison between derived thresholds

6 7 8Time [ms]

Linear

Constant timing jitter

Constant relative timing jitter

Exponential

Heavy-tail

Prapun Suksompong Capacity Analysis of Neurons

Page 28: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

First Jitter: Spike generationSecond and Third JittersThe Threshold Curve

Summary

I Analyze and quantify three sources of timing jitter

I Predict shape of threshold curves

6 7 8Time [ms]

Linear

Constant timing jitter

Constant relative timing jitter

Exponential

Heavy-tail

Prapun Suksompong Capacity Analysis of Neurons

Page 29: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Introduction

Sources of Variability for the ISIs and Derivation of the Threshold

Information-Theoretic Analysis of IF NeuronOPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Conclusion

Prapun Suksompong Capacity Analysis of Neurons

Page 30: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Conditional density Q(t|λ) = fτ |Λ(t|λ)

I We have formula(s) for the threshold curve T (t).

I Assumption: λ stays constant during each ISI.

I Given Poisson input intensity λ, can find the conditionaldensity Q(t|λ) = fτ |Λ(t|λ).

I τ = g(Λ)+jitter.

 

τ  Λ λ  

τ|Λ |λ

Prapun Suksompong Capacity Analysis of Neurons

Page 31: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Optimization 1: Mutual Information

OPT1:sup I (Λ; τ)

where

I (Λ; τ) = E[

logfΛ,τ (Λ, τ)

fΛ(Λ)fτ (τ)

]and the supremum is taken over all possible fΛ(λ).

I Blahut-Arimoto Algorithm (BAA)

Prapun Suksompong Capacity Analysis of Neurons

Page 32: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Exponential Threshold

“Constant-Jitter” Threshold

“Constant-Relative-Jitter” Threshold

C = 5.457 C = 4.931 C = 5.109

(a) (b) (c)

Exponential Threshold

“Constant-Jitter” Threshold

“Constant-Relative-Jitter” Threshold

C = 5.4743 C = 5.2558 C = 4.8279

(a) (b) (c)

Algorithm Exponential Threshold

“Constant Jitter” Threshold

“Constant Relative Jitter” Threshold

Blahut-Arimoto 5.4743 5.2558 4.8279

0 500 1000 1500 20000

2

4

6x 10

-3

f (

)

4 6 8 10 12 140

0.2

0.4

0.6

0.8

t

f (t

)

0 500 1000 1500 20000

2

4

6x 10

-3

f

()

4 5 6 7 80

0.2

0.4

0.6

0.8

t

f (t

)0 500 1000 1500 2000

0

2

4

6x 10

-3

f (

)

4 5 6 7 8 90

0.2

0.4

0.6

0.8

t

f (t

)

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.005

0.01

0.015

0.02

f (

)

2 4 6 8 10 12 14 160

0.2

0.4

0.6

0.8

f ()

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.01

0.02

0.03

f (

)

4.5 5 5.5 6 6.5 7 7.5 80

0.2

0.4

0.6

0.8

f ()

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.01

0.02

0.03

f (

)

4.5 5 5.5 6 6.5 7 7.50

0.2

0.4

0.6

0.8

f ()

Prapun Suksompong Capacity Analysis of Neurons

Page 33: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Capacity-achieving input densities look similar.

0 200 400 600 800 1000 1200 1400 1600 1800 20000

1

2

3

4

5

6x 10

−3

λ

f Λ(λ

)

exponentialconstant jitterconstant relative jitter

Prapun Suksompong Capacity Analysis of Neurons

Page 34: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Input-Intensity Density Approximation

I BAA does not provide anyinsight.

Our simpler formula:

fΛ (λ) ≈ σ0

d

√(c1H (t))3

T (t) c2H2 (t)

∣∣∣∣∣∣t=g(λ)

where g(λ) = E [τ |λ].

Prapun Suksompong Capacity Analysis of Neurons

Page 35: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Input-Intensity Density Approximation

I BAA does not provide anyinsight.

Our simpler formula:

fΛ (λ) ≈ σ0

d

√(c1H (t))3

T (t) c2H2 (t)

∣∣∣∣∣∣t=g(λ)

where g(λ) = E [τ |λ].0 200 400 600 800 1000 1200 1400 1600 1800 2000

0

1

2

3

4

5

6x 10

−3

λ

f Λ(λ

)

exponentialconstant jitterconstant relative jitterapproximation

Prapun Suksompong Capacity Analysis of Neurons

Page 36: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Input-Intensity Density Approximation

I BAA does not provide anyinsight.

Our simpler formula:

fΛ (λ) ≈ σ0

d

√(c1H (t))3

T (t) c2H2 (t)

∣∣∣∣∣∣t=g(λ)

where g(λ) = E [τ |λ]. 101

102

103

10−3

λ

f Λ(λ

)

exponentialconstant jitterconstant relative jitterapproximation

Prapun Suksompong Capacity Analysis of Neurons

Page 37: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Approximation Strategy

Assume that g(Λ) is uniform and then find the corresponding fΛ.

Exponential Threshold

“Constant-Jitter” Threshold

“Constant-Relative-Jitter” Threshold

C = 5.457 C = 4.931 C = 5.109

(a) (b) (c)

Exponential Threshold

“Constant-Jitter” Threshold

“Constant-Relative-Jitter” Threshold

C = 5.4743 C = 5.2558 C = 4.8279

(a) (b) (c)

Algorithm Exponential Threshold

“Constant Jitter” Threshold

“Constant Relative Jitter” Threshold

Blahut-Arimoto 5.4743 5.2558 4.8279

0 500 1000 1500 20000

2

4

6x 10

-3

f (

)

4 6 8 10 12 140

0.2

0.4

0.6

0.8

t

f (t

)

0 500 1000 1500 20000

2

4

6x 10

-3

f (

)

4 5 6 7 80

0.2

0.4

0.6

0.8

t

f (t

)

0 500 1000 1500 20000

2

4

6x 10

-3

f (

)

4 5 6 7 8 90

0.2

0.4

0.6

0.8

t

f (t

)

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.005

0.01

0.015

0.02

f (

)

2 4 6 8 10 12 14 160

0.2

0.4

0.6

0.8

f ()

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.01

0.02

0.03

f (

)

4.5 5 5.5 6 6.5 7 7.5 80

0.2

0.4

0.6

0.8

f ()

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.01

0.02

0.03

f (

)

4.5 5 5.5 6 6.5 7 7.50

0.2

0.4

0.6

0.8

f ()

Prapun Suksompong Capacity Analysis of Neurons

Page 38: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Approximation Strategy

Assume that g(Λ) is uniform and then find the corresponding fΛ.

*

𝑓 𝑔

𝑓𝑔

𝑓𝑁 𝑓

Convolution

For invertible function g , the pdf of Z = g(Λ) is given by

fZ (z) =

∣∣∣∣ d

dzg−1 (z)

∣∣∣∣ fΛ (g−1 (z))

=1

|g ′ (λ)|fΛ (λ) ,

where z = g(λ).Prapun Suksompong Capacity Analysis of Neurons

Page 39: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

OPT2: Energy-Efficient Neuron

I Brains consume 20% of energy consumption for adults and60% for infant [Laughlin and Sejnowski’03].

I Suppose neuron spendsI 1 unit of energy per ms when it is idle, andI e unit of energy per ms when AP is produced.

I e >> 1.

I If the time to the next spike is τ = t, the energy expended is

bo(t) = 1× (t −∆) + e ×∆ = t + (e − 1)∆ = t + r .

where ∆ is the time used to produce a spike.I The value of r depends on the type of neurons under

consideration.

Prapun Suksompong Capacity Analysis of Neurons

Page 40: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Optimization 2: I/E

OPT2:

supI (Λ; τ)

E [bo(τ)]

where the supremum is taken over all possible fΛ(λ).

I Jimbo-Kunisawa algorithm (JKA) maximizes I (Λ;τ)E[b(Λ)] .

I b is a function of input.

I Our bo is a function of output.I We define b(λ) = E [bo(τ)|Λ = λ] and apply JKA.

I Because bo(τ) = τ + r , we have

b(λ) = E [τ |λ] + r = g(λ) + r .

Prapun Suksompong Capacity Analysis of Neurons

Page 41: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Input-Intensity Density Approximation

I Can use the same technique as in OPT1 to do approximationof input-intensity density.

I In stead of uniform density, consider bounded exponentialdensity of the form

f (t; γ, α, β) =γ

e−γα − e−γβe−γt1[α,β] (t) .

Prapun Suksompong Capacity Analysis of Neurons

Page 42: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Input-Intensity Density Approximation

I Can use the same technique as in OPT1 to do approximationof input-intensity density.

I Result:

fΛ (λ) ≈ σ0f (t; γ, g(b), g(a))

√(c1H (t))3

T (t) c2H2 (t)

∣∣∣∣∣∣t=g(λ)

,

where f (t; γ, g(b), g(a)) is the bounded exponential pdf withsupport on the interval [g(b), g(a)] and parameter γ.

Prapun Suksompong Capacity Analysis of Neurons

Page 43: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

0 500 1000 1500 20000

1

2

3

4

5

6x 10

−3

λ

f Λ(λ

)

exponentialconstant jitterconstant relative jitterapproximation

4 6 8 10 12 140

0.1

0.2

0.3

0.4

0.5

t [ms]

f τ(t)

exponentialconstant jitterconstant relative jitter

101

102

103

10−3

λ

f Λ(λ

)

exponentialconstant jitterconstant relative jitterapproximation

4 5 6 7 8 9 10

10−1

t [ms]

f τ(t)

exponentialconstant jitterconstant relative jitter

Prapun Suksompong Capacity Analysis of Neurons

Page 44: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Free Parameters - Revisited

Recall, for example, the differential equation that define our“constant-relative-jitter” threshold:

T ′ (t) = T (t)

(h (t)

H (t)

)−√

T (t)

(1

tσ%time,0

√c2H2 (t)

c1H (t)

).

There are a couple of parameters which we want to revisit.I The constants c1 and c2.

I Embedded in them is the effect of QSF.

I σ%time,0 =σtime,0

t0. What value should we set σtime,0 to be?

I > 10µs.

Prapun Suksompong Capacity Analysis of Neurons

Page 45: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

I By scaling the unit of the voltage, we can make c1 = 1.

I The scaling makes

c2 ∝1

psuccess=

1

1− pfailure

where pfailure is the QSF probability.I pfailure depends on the type of neurons under consideration.

I Let σtime,0 = σ1 and play with it.

Prapun Suksompong Capacity Analysis of Neurons

Page 46: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Average Rates

From our optimization (either OPT1 or OPT2), we get the optimalinput-intensity density fΛ(λ) and spiking-time density fτ (t) for eachvalue of σ1.

I Each value of σ1 gives average arrival rate λ̄in and the averagespiking rate λ̄out.

I λ̄out = 1E[τ ] .

I λ̄in = E [Λ]?

Prapun Suksompong Capacity Analysis of Neurons

Page 47: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Average Afferent Rate

I High value of Λ corresponds to low value of τ .

I Neuron experiences large Λ value for short amount of time.

I λ̄in should be lower than E [Λ].

I Let (Λi , τi ) be the pair of input-intensity and the length of theISI associated with the ith spike.

I The number of input spikes during the ith ISI is a Poissonrandom variable Ni with mean Λiτi .

I The long-term average input rate is then∑ki=1 Ni∑ki=1 τi

=1k

∑ki=1 Ni

1k

∑ki=1 τi

→ E [Λ1τ1]

E [τ1].

I λ̄in = E[Λτ ]E[τ ] .

Prapun Suksompong Capacity Analysis of Neurons

Page 48: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Rate Matching

I On average, our neuron in consideration is bombarded with anaverage input-intensity λ̄in.

I Suppose our neuron is receiving input from n other neurons.Then, on average, each of the sending neurons fire at a rateof 1

n λ̄in.

I Our neuron should also generate spikes at this rate.

I Therefore, we must have

1

nλ̄in = λ̄out.

Prapun Suksompong Capacity Analysis of Neurons

Page 49: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Rate Matching (Con’t)

10 20 30 40 50 60 70 80 90 1000

50

100

150

200

σ1 [microsec]

Ave

rage

spi

king

rat

e [s

pike

s/s]

inputoutput

10 20 30 40 50 60 70 80 90 100

4.8

5

5.2

5.4

σ1 [microsec]

Cap

acity

[bits

]

Prapun Suksompong Capacity Analysis of Neurons

Page 50: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Rate Matching (Con’t)

I Capacity value increases aswe increase the noise levelσ1.

I Larger value of σ1 impliesthreshold decays slower Thisgives larger support for thespiking time.

I Noise is small. Effect ofincreasing the support of theoutput is stronger than theeffect of increased noise.

10 15 20 25 30 35 40 45 50 55 600

50

100

150

200

σ1 [microsec]

Ave

rage

spi

king

rat

e [s

pike

s/s]

inputoutput

10 15 20 25 30 35 40 45 50 55 60

5.2

5.4

5.6

5.8

σ1 [microsec]

Cap

acity

[bits

]

Prapun Suksompong Capacity Analysis of Neurons

Page 51: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

 

1 1.5 2 2.5 3 3.5 4 4.5 515

20

25

30

35

40

45

50

55

60

c2

Opt

imal

rate

[spi

kes

per s

econ

d]

(c) max I/E, r = 100 

exponentialconstant relative jitter

(b) max I/E, r = 400 

(a) max I 

Prapun Suksompong Capacity Analysis of Neurons

Page 52: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Rate Matching (Con’t)

I Firing rate decreases as thefailure probability increases.

I Smaller value of rcorresponds to higher rate.

I As r →∞, the rateconverges to the max-I case.

 

1 1.5 2 2.5 3 3.5 4 4.5 515

20

25

30

35

40

45

50

55

60

c2

Opt

imal

rate

[spi

kes

per s

econ

d]

(c) max I/E, r = 100 

exponentialconstant relative jitter

(b) max I/E, r = 400 

(a) max I 

Prapun Suksompong Capacity Analysis of Neurons

Page 53: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching

Optimal Threshold: c2 = 5 and r = 400

0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

Time [ms]

exponentialconstant relative timing jitter

Prapun Suksompong Capacity Analysis of Neurons

Page 54: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Introduction

Sources of Variability for the ISIs and Derivation of the Threshold

Information-Theoretic Analysis of IF Neuron

Conclusion

Prapun Suksompong Capacity Analysis of Neurons

Page 55: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

Contributions

1. Quantify the amount of timing jitter in neuron.

2. Construct threshold functions.

3. Provide optimal operating points for neurons which are closeto experimentally observed values.

I Formulas to approximate the optimal input-intensity densities.

Prapun Suksompong Capacity Analysis of Neurons

Page 56: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

References I

P. Suksompong and T. Berger.Jitter Analysis of Timing Codes for Neurons with DescendingAction Potential Thresholds.ISIT, 2006.

P. Suksompong and T. Berger.Capacity Analysis of Neurons with Descending ActionPotential Thresholds.In preparation for special issue of IEEE Tran. on Info. Theory,2009.

Prapun Suksompong Capacity Analysis of Neurons

Page 57: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

References II

P. Suksompong and T. Berger.Energy-Efficient Neurons with Descending Action PotentialThresholds.In preparation for Journal of Comp. Neuroscience, 2009.

T. Berger and W.B. Levy.Encoding of excitation via dynamic thresholding.Society for Neuroscience, 2004.

W.B. Levy and R.A. Baxter.Energy-Efficient Neuronal Computation via Quantal SynapticFailures.Journal of Neuroscience, 2002.

Prapun Suksompong Capacity Analysis of Neurons

Page 58: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

References III

W.B. Levy and R.A. Baxter.Energy Efficient Neural CodesNeural Computation, 1996.

Patrick Crotty and William Levy.Biophysical limits on axonal transmission rates in axons.CNS, 2005.

E. Parzen.Stochastic Processes.Holden Day, 1962.

H. Vincent Poor.An introduction to signal detection and estimation (2nd ed.).Springer-Verlag New York, Inc., New York, NY, USA, 1994.

Prapun Suksompong Capacity Analysis of Neurons

Page 59: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

References IV

Dale Purve et al.Neuroscience (3rd ed.).Sinauer Associates Inc., Sunderland, MA USA, 1997.

Prapun Suksompong Capacity Analysis of Neurons

Page 60: Prapun B-Exam

IntroductionSources of Variability and Threshold Derivation

Information-Theoretic Analysis of IF NeuronConclusion

THE END

Prapun Suksompong Capacity Analysis of Neurons