Spatio-Temporal Models for Mental Processes from

fMRIRaghu Machiraju

Firdaus Janoos, Fellow, Harvard MedicalIstavan (Pisti) Morocz, Instuctor, Harvard Medical

PremiseUnderstanding the mind not only requires a comprehension of the workings of low–level neural networks but also demands a detailed map of the brain’s functional architecture and a description of the large–scale connections between populations of neurons and insights into how relations between these simpler networks give rise to higher–level thought

Goals• Understanding the representation of

mental processes in functional neuroimaging– Distributed interactions– Space and time !

• Comparing processes across subjects

• Neurophysiologic interpretability

Outline• fMRI Analysis

• Representations

• Spatio-temporal Models

• Conclusion

What Is fMRI ?• fMRI is a non-invasive tool

for studying brain activity • Spatio-temporal data (4D)• Spatial resolution – mm • Temporal resolutions – secs

• Functional specialization• Classical neuroscience

• Functional integration • Functional and effective

The fMRI Signal• The BOLD Effect –Measure of cerebral metabolism

• Task related• Default-state networks

• Confounds/Nuisance– Random – thermal + quantum mechanical – Structured component

• Distortions, physiological, motion, reconstruction

The BOLD EffectMeasure of “oxygenated blood” in the brain– Volume of deoxyhemoglobin – T2

* weighted EPI sequences

The exact coupling between neuronal activity and the BOLD signal unknown

Linked primarily to metabolic activity at synapsesDepends on rCBF, rBVO2, rCMRO2 The hemodynamic response function is highly variable

fMRI Noise• Acquisition

• Reconstruction

• Magnetic field• Inhomogeneities • Instability

• Physiologic functions• Aliased onto signal

• Head motion• Correlated with the task• Registration / Correction

Classical Pipeline

fMRI Analysis• Functional Localization• Static Activity Maps

• GLM, PCA, ICA, PLS,

• Functional Integration• Functional Connectivity

• CCA, ICA, PCA, DBN• Effective Connectivity

• SEM, DCM, DBN

Typical DCM

Benefits • fMRI provides information about the activity of large

neural assemblies– Static pictures of the foci of activity and the

interconnections

• Mental processes arise from dynamic relationships between the neural substrates– Spatially distributed, temporally transient and occur at

multiple scales of space and time.

• Time resolved analysis– Ordering of information processing

Cascadic Recruitment

State-of-the-ArtJanoos et al., EuroVis2009

Need Decoding !• VOXEL-wise Representations Limited• Dynamic Processes• Distributed Representations Needed – Beyond functional localization• Where vs. how

– Distributed activity and functional interactions

• Pattern Classifiers • Atoms of Thought for Cracking Neural Code

Haxby, 2001

Mitchell, 2008

Challenges• Very controlled experiments with copious training

• General results have not always been positive

• Applications to arbitrary settings ?

• Temporal nature of mental processes

• Neurophysiologic interpretability

• Multi-subject analysis

Inspiration

Lehmann, 1994

Preliminary Results

visuo–spatial working memory

2 Patients

Functional Networks

Functional Connectivity Estimation

Gaussian smoothing

HAC until f ≈0.25N

Cluster-wise Correlation Estimation and Shrinkage

Voxel-wise Correlation Estimation

Functional Distance

Zt – activation patternsf - transportation

Cost Metric

Functional Distance

t1 t2

t3

Algorithm

Mental Arithmetic• Involves basic manipulation of number and

quantities

• Magnitude based system – bilateral IPS

• Verbal based system – left AG

• Attentional system – ps Parietal Lobule

• Other systems – SMA, primary visual cortex, liPFC, insula, etc

Paradigm

Clustering in Functional Space

10

0s 4s 8s0s 4s 8s

Bra

in S

tate

Lab

el

5

0

10

5

0

Spatial Maps10 same as 8

8 auditory cortices

6 , judgment

5 Frontal, parietal lobes

3 visual size estimation

1 Visual Cortex

0+5.0 -5.00s 4s 8s

10

6

1

8

7

3

0

5

4

9

2

Critique• No neurophysiologic model– Point estimates– Hemodynamic uncertainty – Temporal structure

• Functional distance - an optimization problem– No metric structure– Expensive !

Functional Distance

Cost Metric

Cost MetricDistortion minimizing

Feature Space Φ

Orthogonal Bases Graph Partitioning

Normalized graph Laplacian of F

Feature Space Φ

Feature SelectionY

Φ

Rtimes

Resampling with Replacement

Basis Vector φ(l,m) Computation

Bootstrap Distribution of Correlations ρ (l,m)

Feature SelectionRetain φ(l,m) if Pr[ρ (l,m) ≥ τΦ] ≥ 0.75

Functional Network Estimation

State-Space Model

xt

zt

yt

. . .

. . .

xt+1

zt+1 zt+2 zt+L

. . .

T

yt+1 yt+2 yt+L

xt+2 xt+L

μγ σγγ

h

μk ΣkK

α,π

Σε

Janoos et al., MICCAI 2010

(Reduced) State-Space Model

xt

yt

xt+1

yt+1

xt+2

yt+2

xt+L-1

yt+L

…

…T

Σε

μγ σγ

μk ΣkK

γ

h

α, π

Model Size Selection• Typically strike a balance between

model complexity and model fit

• Information theoretic or Bayesian criteria– Notion of model complexity

• Cross-validation– IID Assumption

Maximally Predictive Criteria• Multiple spatio-temporal patterns in

fMRI– Neurophysiological

• task related vs. default networks– Extraneous

• Breathing, pulsatile, scanner drift• Select a model that is maximally

predictive with respect to task– Predictability of optimal state-sequence

from stimulus, s

DyscalculiaDifficulty in learning arithmetic that cannot be explained by mental retardation, inappropriate schooling, or poor social environment

• Core conceptual deficit dealing with numbers

• Very common : 3-6% of school-age children• Heterogeneous

Paradigm

Results Self – same subjectCross – train on one subject and predict on another

Comparing Modelsst

xt

zt

yt

st+1

xt+1

zt+1

yt+1

ut+2

xt+2

zt+2

yt+2

st+L-1

xt+L-1

zt+L-1

yt+L-1

…

…

…

…

T

W

μk Σk

K

h

Σε

λW

μh Σh

Subject 1

.

.

.

st

xt

zt

yt

st+1

xt+1

zt+1

yt+1

ut+2

xt+2

zt+2

yt+2

st+L-1

xt+L-1

zt+L-1

yt+L-1

…

…

…

…

T

W

μk Σk

K

h

Σε

λW

μh Σh

Subject 42

st

xt

zt

yt

st+1

xt+1

zt+1

yt+1

ut+2

xt+2

zt+2

yt+2

st+L-1

xt+L-1

zt+L-1

yt+L-1

…

…

…

…

T

W

μk Σk

K

h

Σε

λW

μh Σh

Subject 2

fMRI Data

xt xt+1 xt+2 xt+L-1…

xt xt+1 xt+2 xt+L-1…

xt xt+1 xt+2 xt+L-1…

1 2 … 41 421 10.00 8.94 … 8.50 5.402 8.94 10.00 … 1.54 0.29⁞ ⁞ ⁞ ⁞ ⁞

41 8.50 1.54 … 10.00 3.9542 5.40 0.29 … 3.95 10.00

Φ1

Φ2

Φ42

MDS Plot

MDS Plot

Drawbacks• Approximations in the model– Elimination of the activity pattern layer– Spatially unvarying hemodynamics

• Unsupervised approach– No explicit link to the experiment–May not necessarily learn relevant

patterns

Semi-supervised Approach• Loose dependency between stimulus

and signal– Not preclude discovery of un-modeled

effects– Stabilize estimation

• Generalizable to unconstrained designs

• Functionally well-defined representation

The Modelst

xt

zt

yt

st+1

xt+1

zt+1

yt+1

ut+2

xt+2

zt+2

yt+2

st+L-1

xt+L-1

zt+L-1

yt+L-1

…

…

…

…

T

W

μk Σk

K

h

Σε

λW

μh Σh

Janoos et al., IPMI 2011Janoos et al., NeuroImage 2011

EM Algorithm

Mean Field Approximation

Estimation

Model Estimation

State Sequence Estimation

Φ Feature-Space Transformation

y

Until convergence

θ

Until convergence

s

K, λWError Rate

HyperparameterSelection

x

YfMRI Data

Feature-space basis

E-stepCompute q(n)(x,z) from p(y,z,x|θ(n))

M-stepEstimate θ(n+1) : L(q(n), θ(n+1)) > L(q(n), θ(n))

E-stepCompute q(n)(z) from p(z| y,x(n),θ)

M-stepx(n+1) = argmax L(q(n), x)

Stimulus Parameters

Hyperparameters

DyslexiaSelective inability to build a visual representation of a word, used in subsequent language processing, in the absence of general visual impairment or speech disorders

• Affects 5-10% of the population• Spelling, phonological processing, word retrieval• Disorder of the visual word form system• Multiple varieties– Occipital, temporal, frontal, cerebellum

Paradigm

Comparative Results

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

SVM SSM:FULL SSM:PH SSM:NONE

FS:Φ

FS:PCA-NONE

FS:PCA-PH

FS:PCA-FULL

Error for PH with FS:PCA-PH

Overall Results

1

2

3

Spatial Maps

Hemodynamic Responses

Motor Cortex

Intra Parietal Sulcus

MDS Plots

MDS Plots

Control MaleControl Female

Dyslexic FemaleDyslexic Male

Dyscalculic MaleDyscalculic Female

Phase 1

Phase 2

Phase 1: Product Size

Phase 2: Problem Difficulty

MDS Plots (2)

Conclusion• Process model for fMRI – Spatial patterns and the temporal structure– Identification of internal mental processes

• Neurophysiologically plausible– Test for the effects of experimental

variables– Parameter interpretation

• Comparison of mental processes– Abstract representation of patterns

Thank You