A Dynamical Pattern Recognition Model of Gamma Activity in

Auditory Cortex

A Dynamical Pattern Recognition Model of Gamma Activity in

Auditory Cortex

Will PennyWill Penny

Wellcome Trust Centre for Neuroimaging,Wellcome Trust Centre for Neuroimaging,University College London, UKUniversity College London, UK

Centre for Integrative Neuroscience and Neurodynamics,University of Reading, 17th March 2010

GammaActivity(>30Hz)

ComputationalNeuroscience

Imaging Neuroscience

SpikeTimeDependentPlasticity

BOLDsignal

BindingProblem

Localisation

Miller, PLOS-CB,2009

Goense et alCurr Biol, 2008

Singer,Neuron, 1999

Overview

• Bandpass filtering and feature detection• Spike Frequency Adaptation• Spike Time Dependent Plasticity • Neuronal Synchronization• Sparse Coding

Hopfield-Brody Model

Levels of Analysis Occurrence Times for Speech Recognition

Forward model of imaging data and stimulus labels

Overview

• Bandpass filtering and feature detection• Spike Frequency Adaptation• Spike Time Dependent Plasticity • Neuronal Synchronization• Sparse Coding

Hopfield-Brody Model

Levels of Analysis Occurrence Times for Speech Recognition

Forward model of imaging data and stimulus labels

Bandpass filters

Allen (1985) Cochlear Modelling IEEE ASSP

Onset Cells, Offset Cells, etc.

Hromadka et al, PLOS B, 2008Cell attached recordings from A1 in rat

Onset and offset cells with frequency adaptation

Hromadka et al, PLOS B, 2008

Overview

• Bandpass filtering and feature detection• Spike Frequency Adaptation• Spike Time Dependent Plasticity • Neuronal Synchronization• Sparse Coding

Hopfield-Brody Model

Levels of Analysis Occurrence Times for Speech Recognition

Forward model of imaging data and stimulus labels

Spike Time Dependent Plasticity

Abbott and Nelson. Nat Neuro, 2000

Pre

Post

25ms

Neuronal Synchronization

Frequencies need to be similar (Kuramoto)

With fast connections excitation causes sync (Gap Junctions)

With slow connections inhibition causes sync (Chemical synapses, PING)

Hopfield Brody – balanced excitation and inhibition

See Bartos, Van Vreeswijk, Ermentrout, Traub ….

Hopfield and Brody, PNAS 2001

Hopfield Brody Model

0.2 0.4 0.6 0.8 1-0.5

0

0.5Audio waveform

time (s) time (s)

freq

uenc

y (H

z)

Audio spectrogram

0.2 0.4 0.6

1000

2000

3000

0.2 0.4 0.6 0.8 10

50

time (s)

Spikes

0.2 0.4 0.6 0.8 10

1

time (s)

In Currents

0.2 0.4 0.6 0.8 1

0.20.40.60.8

time (s)

LFP

time (s)

freq

uenc

y (H

z)

LFP Spectrogram

0 0.5 1

20

40

60

80

“Add”

0.2 0.4 0.6 0.8 1-0.5

0

0.5Audio waveform

time (s) time (s)

freq

uenc

y (H

z)

Audio spectrogram

0.2 0.4 0.6

1000

2000

3000

0.2 0.4 0.6 0.8 10

50

time (s)

Spikes

0.2 0.4 0.6 0.8 10

1

time (s)

In Currents

0.2 0.4 0.6 0.8 1

0.4

0.6

time (s)

LFP

time (s)

freq

uenc

y (H

z)LFP Spectrogram

0 0.5 1

2040

60

80

“Add”

Word 1 Word 2

30 frequency bands x 3 types (onset,offset,peak) x 10 adaptation times=900 cells

Each word coded for by activity in an ensemble of 45 cells

Frequencies near recognition time point

Sparse Coding

Hromadka et al, PLOS B, 2008

fb

tb

k=1 k=2

j=1 j=1

j=2 j=2

Time

Frequency

fmax

Spike Time Dependent Plasticity

Overview

• Bandpass filtering and feature detection• Spike Frequency Adaptation• Spike Time Dependent Plasticity • Neuronal Synchronization• Sparse Coding

Hopfield-Brody Model

Levels of Analysis Occurrence Times for Speech Recognition

Forward model of imaging data and stimulus labels

Levels of Analysis

Algorithmic level – what algorithm is being used ?

Example – Fourier Analysis

Implementation level – how is it implemented ?

Example – FFT if you have digital storage, adders, multipliers Holography if you have a laser and photographic plates

See eg. Marr and Poggio, AI Memo, 1976

Levels of Analysis

Algorithmic level – what algorithm is being used ?

Similarity of Occurrence Times

Implementation level – how is it implemented ?

Transient Synchronisation

Recognising Patterns

Bandpassfilteringat multiple spatial scales

Spatial configurationof Features

Leveldetection

Bandpassfilteringat multiple temporal scales

Temporal configurationof Features

Leveldetection

Fukushima, Rolls etc.

Overview

• Bandpass filtering and feature detection• Spike Frequency Adaptation• Spike Time Dependent Plasticity • Neuronal Synchronization• Sparse Coding

Hopfield-Brody Model

Levels of Analysis Occurrence Times for Speech Recognition

Forward model of imaging data and stimulus labels

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6-0.5

0

0.5

1

x

t

F (

Hz)

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.50

1000

2000

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.60

100

200

300

f (H

z)Occurrence Times

y=[t1, t2, t3, ……, t45]

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6-0.5

0

0.5

1

x

t

F (

Hz)

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.50

1000

2000

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.60

100

200

300

f (H

z)Autoregressive features

K frames

)()(...)2()1()( 21 teptxatxatxatx p EachFrame:

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

8

9

10

0

1

2

3

4

5

6

7

8

9

Confusion matrix from one cross-validation run

Truth

Cla

ssifi

catio

n

HB spokendigit database

FirstNearestNeighbourClassifier

Occurrence TimesAutoregressive Parameters

Single shot learning results

Results on single subject

Overview

• Bandpass filtering and feature detection• Spike Frequency Adaptation• Spike Time Dependent Plasticity • Neuronal Synchronization• Sparse Coding

Hopfield-Brody Model

Levels of Analysis Occurrence Times for Speech Recognition

Forward model of imaging data and stimulus labels

ECOG recordings over fronto-temporal cortex

Canolty et al. Front. Neuro, 2007

Subject listened to words and “nonwords”

• Task: press button if word is a persons name (this data not analysed)

• Each nonword was created by taking a word, computing the spectrogram and removing ripple sound components corresponding to formants. The spectrogram was then inverse transformed (Singh and Theunissen, 2003)

• Each nonword matched one of the words in duration, intensity, power spectrum, and temporal modulation.

Time (ms)

Fre

quen

cy (

Hz)

Words

-200 0 200 400 600 80060

80

100

120

140

160

2.5

3

3.5

4

4.5

5

5.5

Time (ms)

Fre

quen

cy (

Hz)

NonWords

-200 0 200 400 600 80060

80

100

120

140

160

2.5

3

3.5

4

4.5

5

5.5

Time (ms)

Fre

quen

cy (

Hz)

Difference

-200 0 200 400 600 800

20

40

60

80

100

120

140

160

-0.2

0

0.2

0.4

0.6

0.8

Time (ms)

Fre

quen

cy (

Hz)

T-score

-200 0 200 400 600 800

20

40

60

80

100

120

140

160

-2

0

2

4

6

8

10

12

14

ECOG recordings over STS

u y

x

u yu y

Encoding

Symmetric

Decoding

u yStimulusLabels

LocalField

x

tk

s

f(x,t)

AuditoryInput Pattern

BandpassFilters

Onset, offsetand peak responsetimes

Input-dependentFrequencies

Weakly-coupledOscillators Pattern

Recognition

FeatureExtraction

WCO-TS Model

)]()(sin[),()( ttwtxft ijj

ijii

Input-pattern dependent frequencies

LateralConnectivity:Uniform (A)

Sync is stablestate

j

j tty )(cos)(

LFP

WCO-TS

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

20

40

60

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

20

40

60

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2-20

0

20

f(x,t)

cos (t)

y(t)

fb

tb

k=1 k=2

j=1 j=1

j=2 j=2

Time

Frequency, fi(x,t)

fmax

Minimal model with four parameters: ={A, fmax, fb, tb }

Time (ms)

Fre

quen

cy (

Hz)

Words

-200 0 200 400 600 80060

80

100

120

140

160

2.5

3

3.5

4

4.5

5

5.5

Time (ms)

Fre

quen

cy (

Hz)

NonWords

-200 0 200 400 600 80060

80

100

120

140

160

2.5

3

3.5

4

4.5

5

5.5

Time (ms)

Fre

quen

cy (

Hz)

Difference

-200 0 200 400 600 800

20

40

60

80

100

120

140

160

-0.2

0

0.2

0.4

0.6

0.8

Time (ms)

Fre

quen

cy (

Hz)

T-score

-200 0 200 400 600 800

20

40

60

80

100

120

140

160

-2

0

2

4

6

8

10

12

14

ECOG recordings over STS

Word

30 frequency bands x 3 types (onset,offset,peak) x 10 adaptation times=900 cells

Each word coded for by activity in an ensemble of 45 cells

Frequencies near recognition time point

For each of 96trials look at response to wordversus nonword

Computational saving:only look at those cells activated by both word and nonword stimuli

Model fitting

• EN: ECOG spectra-model spectra

• EC: Word versus non word

)exp()|( ECENyp

Uniform priors, Metropolis Hastings estimation of posterior densities

Training Testing

Minimal Model

Testing Training

Augmented Model

SummaryLevels of Analysis Occurrence Times for Speech Recognition

Forward model of imaging data and stimulus labels

FUTURE:

Lack of direct evidence for TS mechanism

Can STDP rules synchronize multiple spikes ?

Have modelled activity in single region only.

GammaActivity(>30Hz)

ComputationalNeuroscience

Imaging Neuroscience

SpikeTimeDependentPlasticity

BOLDsignal

BindingProblem

Localisation

Miller, PLOS-CB,2009

Goense et alCurr Biol, 2008

Singer,Neuron, 1999

Melissa Zavaglia & Mauro UrsinoDEIS, Cesena, Italy

LIF network simulations

Rich Canolty & Bob Knight,UCSF, San Francisco.

ECOG data and analysis

Tom Schofield & Alex Leff,UCL, London.

Cognitive neuroscience of language

Acknowledgements

time (s)

freq

uenc

y (H

z)

Word 6, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 2

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 3

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 4

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 5

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 6

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 7

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 8

0 0.5 1

20406080

time (s)fr

eque

ncy

(Hz)

Word 6, Ex 9

0 0.5 1

20406080

Sensitivity

FN

time (s)

freq

uenc

y (H

z)

Word 1, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 2, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 3, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 4, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 5, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 6, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 7, Ex 1

0 0.5 1

20406080

time (s)

freq

uenc

y (H

z)

Word 8, Ex 1

0 0.5 1

20406080

time (s)fr

eque

ncy

(Hz)

Word 9, Ex 1

0 0.5 1

20406080

Specificity

FP

Optical Imaging

Harrison et al. Acta Oto, 2000