Mind Reading with fMRI

55
Mind Reading with fMRI Ken Norman Department of Psychology Princeton University May 1, 2007

description

Mind Reading with fMRI. Ken Norman Department of Psychology Princeton University May 1, 2007. Brain Scanning. Today’s topic: Applying pattern classifiers to brain scanning data, to decode the information represented in a person’s brain at a particular point in time - PowerPoint PPT Presentation

Transcript of Mind Reading with fMRI

Page 1: Mind Reading with fMRI

Mind Reading with fMRI

Ken Norman

Department of Psychology

Princeton University

May 1, 2007

Page 2: Mind Reading with fMRI

Brain Scanning

• Today’s topic: Applying pattern classifiers to brain scanning data, to decode the information represented in a person’s brain at a particular point in time

• This is NOT the standard approach

• Standard approach:

• Stick someone in the scanner• Have them perform a cognitive task• Explore which brain regions are engaged by the

cognitive task

Page 3: Mind Reading with fMRI

Brain Scanning

• If you’re interested in memory retrieval:

• Scan people while they’re retrieving memories• Scan people during a control condition

• Look at which brain regions respond differentially

• This approach has been very productive for cognitive neuroscience

Page 4: Mind Reading with fMRI

Brain Scanning

• Alternative approach to analyzing brain scanning data:

• Use pattern classification algorithms, applied to distributed patterns of neural activity, to identify the neural signatures of particular thoughts and memories

• Once we have trained the classifier to recognize a particular thought, we can use the classifier to track the comings and goings of those thoughts over time

Page 5: Mind Reading with fMRI

Motivation

• Why pattern classification?

• Reason #1: Improve the interface between fMRI and cognitive theories

• Cognitive neuroscientists have developed very detailed theories of how information is processed in the brain

• What information is represented in different brain structures?

• How is it represented?• How is that information transformed at different stages of processing?

• To directly test these theories, we need a way of decoding the informational contents of the subject’s brain state

Page 6: Mind Reading with fMRI

Motivation

• Reason #2: We aren’t doing as good a job of “data mining” fMRI data as we could...

• We collect several GB of information from each subject

• There is a lot of information about subjects’ thoughts buried in these big data files; the challenge is how to extract this information

• Machine learning researchers have developed tremendously powerful algorithms for extracting meaningful regularities from large data sets

• These algorithms are not routinely used in fMRI data analysis…

Page 7: Mind Reading with fMRI

Outline• 3 minute overview of functional MRI

• Brief overview of existing research on fMRI pattern classification

• Technical challenges & machine learning issues

Page 8: Mind Reading with fMRI

Brain Scanning 101

Page 9: Mind Reading with fMRI

Brain Scanning 101• How do we image neural activity with functional MRI?

• Brain regions that are active use up more metabolic resources

• In particular, they use up more oxygen from the blood

• The MRI machine can be tuned to detect the difference between oxygenated and deoxygenated blood

• By looking at which brain areas have deoxygenated vs. oxygenated blood, we can get a sense of which brain areas are active at a particular moment

Page 10: Mind Reading with fMRI

Brain Scanning 101• it takes approx. 2 seconds for the MRI machine to take a snapshot of blood flow (across the entire brain)

Page 11: Mind Reading with fMRI

fMRI images• Big cube, made out of a grid of little cubes

– Pixel = one square in a 2D grid (“picture element”)

– Voxel = one of the tiny little cubes in an fMRI image (like a volumetric pixel)

• Voxels are approx. 3 millimeters on each side• Neuron size ~ 10 micrometer

• Each voxel reflects the aggregate activity of a very large number of neurons• We aren’t directly measuring activity, we are measuring blood flow!• Blood flow response is smeared out in time (peak response = ~6 sec after neural activity)

Page 12: Mind Reading with fMRI

Patterns in the brain• Key idea: Cognitive states correspond to distributed patterns of brain activity

• What do these “patterns in the brain” look like?

Page 13: Mind Reading with fMRI

The Eight Categories Study

(Haxby et al. 2001)

The Eight Categories Study

(Haxby et al. 2001)

Faces

Cats

Scissors

Chairs

Houses

Bottles

Shoes

Scrambled Pictures

slides courtesy of Jim Haxby

Page 14: Mind Reading with fMRI

0

10

20

30

40

50

60

70

80

90

100

Accuracy of Category Identification

Accuracy of Category Identification

Identification Accuracy ± SE

Chance

Overall Accuracy = 96%

slides courtesy of Jim Haxby

Page 15: Mind Reading with fMRI

Our Studies• We set out to extend the basic pattern classification method

• The brain patterns from the Haxby study correspond to several minutes’ worth of brain activity

• We wanted to see if we could classify cognitive states based on single brain images (reflecting ~2 seconds’ worth of neural activity)

Page 16: Mind Reading with fMRI

Pattern Classification Method• General approach: Say that we want to be able to track the presence of two different cognitive states in the subject’s brain (e.g., viewing shoes vs. bottles) using fMRI

Page 17: Mind Reading with fMRI

Pattern Classification Method

1. Acquire brain data while the subject is thinking about shoes or bottles

Page 18: Mind Reading with fMRI

Pattern Classification Method

1. Acquire brain data

2. Convert each functional brain volume (~ 2 seconds worth of data) into a vector that reflects the pattern of activity across voxels at that point in time.

We typically do some kind of feature selection to cut down on the number of voxels

Page 19: Mind Reading with fMRI

Pattern Classification Method

1. Acquire brain data

2. Generate brain patterns

3. Label brain patterns according to whether the subject was viewing shoes vs. bottles (adjusting for lag in the blood flow response)

Page 20: Mind Reading with fMRI

Pattern Classification Method

1. Acquire brain data

2. Generate brain patterns

3. Label brain patterns

4. Train a classifier to discriminate between bottle patterns and shoe patterns

Page 21: Mind Reading with fMRI

Simple Neural Network Classifier (Logistic Regression)

• To estimate how much subjects are thinking about bottles, compute a weighted sum of voxel activity values; do the same for shoes

• Apply decision rule (e.g., sigmoid function)

• To train the classifier, we use a learning algorithm that sets the weights to maximize decision performance (e.g., backpropagation)

Output layer

Bottle Shoevs

Input layer (voxels)

Page 22: Mind Reading with fMRI

Pattern Classification Method

1. Acquire brain data

2. Generate brain patterns

3. Label brain patterns

4. Train the classifier

5. Apply the trained classifier to new brain patterns (not presented at training).

Page 23: Mind Reading with fMRI

Free Recall & Mental Time Travel (Polyn et al., 2005)

• How do we selectively retrieve memories from a particular event?

• Intuitively: We try to recapture our mindset from that event

• Concretely: We try to make our brain state during recall resemble our brain state during the original event

• “Mental Time Travel”

• Goal of the study: Use fMRI pattern-analysis to image this process of mental time travel as it happens...

Page 24: Mind Reading with fMRI

Imaging Mental Time Travel (Polyn et al., 2005)• Memory experiment: Subjects study 3 types of

stimuliJack Nicholson Giza pyramids flask

• Recall test: Recall items from all 3 categories, in any order

• Hypothesis: To recall a particular category, subjects try to recapture their mindset from the study phase

• In concrete terms: Subjects try to make their brain state at test resemble their brain state when they were studying that category

• If subjects succesfully recapture their brain state from the study phase, this will trigger recall of specific studied items...

Page 25: Mind Reading with fMRI

Analysis strategy• Step 1: Feed fMRI data from the study phase into a pattern

classification algorithm

• Train the pattern classifier to recognize the brain patterns associated with studying faces vs. locations vs. objects

Page 26: Mind Reading with fMRI

Neural network classifier

• Mapping from voxel activity values to output units (one per category)

Page 27: Mind Reading with fMRI

Analysis strategy• Step 2: Apply the trained classifier to brain data from the

retrieval phase

• Use the classifier to track, second-by-second, how well the subject’s brain state at retrieval matches their brain state when they were studying faces vs. locations vs. objects

Page 28: Mind Reading with fMRI

Predictions

• As subjects try recall faces, locations, and objects, their brain state should come into alignment with the brain states associated with studying faces, locations, and objects

• This neural measure of category-specific “mental reinstatement” should be predictive of recall

Page 29: Mind Reading with fMRI

Final free recall - classifier outputm

atch

to

face

stu

dy c

onte

xt

mat

ch t

o lo

catio

n st

udy

con

text

mat

ch t

o ob

ject

stu

dy

cont

ext

Classifier traces for Subject 9 during final free recall.

Page 30: Mind Reading with fMRI

Final free recall - classifier output

Classifier traces for Subject 9 during final free recall.

mat

ch t

o fa

ce s

tudy

con

text

mat

ch t

o lo

catio

n st

udy

con

text

mat

ch t

o ob

ject

stu

dy

cont

ext

Page 31: Mind Reading with fMRI

Other findings• Kamitani & Tong (2005): decode the orientation of a striped pattern that is being viewed by the subject (accurate to within 20 degrees)

Page 32: Mind Reading with fMRI

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Page 33: Mind Reading with fMRI

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Page 34: Mind Reading with fMRI

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Page 35: Mind Reading with fMRI

2006 Pittsburgh competition• Subjects were scanned while they watched 3 episodes of “Home Improvement”

• Time-varying ratings obtained for “amusement”, “food”, “tools”, “faces”...

• Goal: predict ratings using brain data

• Train a classifier using brain data + ratings from 2 episodes

• Then, feed the trained classifier the brain data from the 3rd episode and use the classifier to predict (in a second-by-second fashion) the subject’s feature ratings

Page 36: Mind Reading with fMRI

2006 competition• some representative correlation values:

• Amusement: .46• Faces: .67• Language: .69• Laughter: .58• Motion: .49• Music: .76• Tools: .62

Page 37: Mind Reading with fMRI

2007 Pittsburgh competition

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

• www.braincompetition.org

Page 38: Mind Reading with fMRI

Interim Summary

• By applying classifiers to fMRI data, we can derive a time-varying estimate of the subject’s cognitive state, that relates in a meaningful way to their behavior

• Technical challenges

Page 39: Mind Reading with fMRI

Technical Challenges

• From the perspective of machine learning, fMRI classification is a particularly difficult problem (Mitchell et al., 2004, Machine Learning)

• Big patterns• Noisy patterns• Relatively few patterns

• What can we do to improve classification?

Page 40: Mind Reading with fMRI

Classifiers

• We have tried lots of classifiers

– Neural network, correlation-based classifiers, support vector machines, Gaussian Naive Bayes, boosting, k-nearest-neighbor, linear discriminant analysis...

• The exact classifier that we use doesn’t seem to matter (much); nonlinear classifiers do not systematically outperform linear classifiers...

• Regularization helps (e.g., ridge regression outperforms normal regression)

Page 41: Mind Reading with fMRI

Feature Selection

• Getting rid of noisy voxels greatly helps performance

• Standard method:

• Run a voxel-wise omnibus ANOVA on the conditions of interest (e.g., face vs. location vs. object)

• Get rid of voxels that don’t vary significantly across conditions

Page 42: Mind Reading with fMRI

Feature Selection

• This ANOVA method helps, but it has several problems

• Main benefit of linear classifiers is that they can aggregate weak signals across voxels

• In light of this, it seems like a bad idea to discard individual voxels just because the voxel’s signal is weak...

Page 43: Mind Reading with fMRI

Feature Selection

• What we really want to do is to come up with multivariate means of voxel selection

• we want to select sets of voxels that in aggregate carry useful information

• Promising approach: Searchlights (Kriegeskorte et al., 2006, PNAS)

Page 44: Mind Reading with fMRI

Dimensionality Reduction

• We are also exploring different methods of re-coding the data

• There is extensive redundancy across voxels (esp. spatially proximal voxels)

• Is there a more efficient way to represent the input (i.e., with fewer dimensions)

• manifold learning• Spatial wavelet decomposition• ICA

Page 45: Mind Reading with fMRI

Dimensionality reduction algorithms

• Generative models (David Weiss & David Blei)

• Each brain state is made of a linear combination of “neural topics”

• Each topic = a pattern of voxel activity across the whole brain (positive and negative values are OK)

• To generate a brain state from topics, multiply each topic by a positive value

• Topics are constrained to be spatially sparse (L2 regularization; trying L1 also)

Page 46: Mind Reading with fMRI

Next steps

• We know a lot about the brain (in general), the fMRI response, and cognition that we are not telling the classifier…

• Currently: Each brain pattern is treated as a distinct observation

• In actuality: There is massive correlation between adjacent time points

• Knowing the information represented at time n tells you a lot about the information represented at time n + 1

Page 47: Mind Reading with fMRI

Next steps

• In addition to temporal correlation, there is extensive spatial correlation

• Nearby voxels tend to represent similar things

• One way to address this issue is by spatially smoothing the data (averaging together activity from nearby voxels)

• However, you can lose information this way

• A more sophisticated approach would be to directly measure pairwise correlations between voxels and incorporate this information in the model

Page 48: Mind Reading with fMRI

Next steps

• Currently, our analyses are focused on single subjects

• Is there some way to leverage data from other subjects to help with classification

• If you run 10 subjects in the Haxby 8-category experiment, none of the subjects will have the exact same “shoe” representation, but the shoe representations are not random either

• It might be possible to draw on data from other subjects to set priors on which voxels will be involved in representing shoes

Page 49: Mind Reading with fMRI

Next steps

• Also, there is an enormous body of evidence relating to which brain structures are involved in a given cognitive task

• “face area”, “place area”

• We can use this information to set priors on voxel weights in the classification process

Page 50: Mind Reading with fMRI

Next steps

• The cognitive states that we are trying to classify often have a hierarchical structure

• How you represent a stimulus depends on the task that you are performing

• Informing the classifier about this hierarchical structure should boost classification

Page 51: Mind Reading with fMRI

Next steps

• Different tasks (dangerous/safe, land/water) have different neural signatures

• If we can detect the neural signatures of these tasks, we can conditionalize the classifer on which task representation is present in the subject’s head

Page 52: Mind Reading with fMRI

Next steps

• Lots of potential constraints

• Temporal autocorrelation• Correlation between nearby voxels• Data from other subjects in the same experiment

• Data from other experiments• Hierarchical structure of cognitive states

• How do we inform the classifier of these constraints?

• Graphical models should provide a way of doing this….

Page 53: Mind Reading with fMRI

Summary

• By applying pattern classification algorithms to neuroimaging data, we can extract a tremendous amount of information regarding what subjects are thinking, and how subjects’ thoughts evolve over time

• Plenty of room for improvement...

• Solving these problems will require meaningful contributions from several disciplines: Cognitive psychology, neuroscience, machine learning, engineering, signal processing, statistics, and mathematics...

Page 54: Mind Reading with fMRI

Computational Memory Lab

• Michael Bannert• Melissa Carroll• Denis Chigirev• Greg Detre• Chris Moore• Ehren Newman• Joel Quamme• Susan Robison• Per Sederberg• Matt Weber• David Weiss• And many others…http://compmem.princeton.edu

Princeton Colleagues• David Blei (Comp. Sci.)

• Matt Botvinick (PSY)• Jon Cohen (PSY)• Ingrid Daubechies (Math)

• Jim Haxby (PSY)• Fei-Fei Li (Comp. Sci.)

• Dan Osherson (PSY)• Peter Ramadge (EE)• Rob Schapire (Comp. Sci.)

• Greg Stephens (Physics)

Page 55: Mind Reading with fMRI

my email: [email protected]

Princeton Multi-Voxel Pattern Analysis Toolkit

currently in public beta-testing:

www.csbmb.princeton.edu/mvpa

NiAM (NeuroImaging Analysis Methods) group

meets Fridays 2pm