Multivoxel Pattern Analysis (MVPA) in fMRI settings : Fundamentals & Case of study
Advance fMRI analysis: MVPA Sara Fabbri Brain and Mind Institute Western University.
85
Advance fMRI analysis: MVPA http://www.fmri4newbies.com/ Sara Fabbri Brain and Mind Institute Western University
-
Upload
piers-blake -
Category
Documents
-
view
215 -
download
0
Transcript of Advance fMRI analysis: MVPA Sara Fabbri Brain and Mind Institute Western University.
Advance fMRI analysis: MVPA
http://www.fmri4newbies.com/
Sara FabbriBrain and Mind Institute
Western University
Overview
A. Why MVPA?
B. How MVPA1. MVPA classifier
2. MVPA correlation
3. Representational similarity approach
C. “Mind-reading”
on
set
off
set
Picture number
Picture
Raster plots for various trials (from top to bottom)
Firing rate
Quiroga et al., 2005, Nature
Response of a neuron from the human medial temporal lobe
Quiroga et al., 2005, Nature
Response of a neuron from the human medial temporal lobe
Quiroga et al., 2005, Nature
Selective response to Jennifer Aniston pictures
Neuron 1“likes”
Jennifer Aniston
Response of a neuron from the human medial temporal lobe
Fir
ing
Rat
e
Fir
ing
Rat
e
Fir
ing
Rat
e
Neuron 1“likes”
Jennifer Aniston
Neuron 2“likes”
Julia Roberts
Neuron 3“likes”
Brad Pitt
Fusiform Face Area (FFA)
3 mm
3 mmlowactivity
highactivity
3 mm
3 mm
A voxel might contain millions of neurons, so the fMRI signal represents the population activity
fMRI spatial resolution: 1 voxel
3 mm
Fir
ing
Rat
e
Fir
ing
Rat
e
Fir
ing
Rat
e
Act
iva
tion
Neuron 1“likes”
Jennifer Aniston
Neuron 2“likes”
Julia Roberts
Neuron 3“likes”
Brad Pitt Even though there are neurons tuned to each actor, the population as a whole shows no preference
Act
iva
tion
Limitation of Subtraction Logic
>
Subtraction
Act
iva
tion
No preference
Are there neurons that prefer Jennifer to Julia in the voxel?
Two Techniques with “Subvoxel Resolution”
• “subvoxel resolution” = the ability to investigate coding in neuronal populations smaller than the voxel size being sampled
1. Multi-Voxel Pattern Analysis (MVPA or decoding or “mind reading”)
2. fMR Adaptation (or repetition suppression or priming)
3 mm
3 mmlowactivity
highactivity
fMRI spatial resolution: 1 voxel
3 mm
3 mm
3 mm
lowactivity
highactivity
Region Of Interest (ROI): group of voxels
Standard fMRI Analysis
trial 1
trial 3
trial 2
trial 1
trial 2
trial 3
FACES HOUSES
AverageSummedActivation
Spatial Smoothing
• Application of a Gaussian low-pass spatial filter to fMRI data resulting in spreading the activation across adjacent voxels
• Why?– increase signal-to-noise– facilitate intersubject averaging
No smoothing 4 mm Full-width-Half-Maximum
(FWHM)
7 mm FWHM 10 mm FWHM
Effect of Spatial Smoothing:reduce spatial resolution
Patterns carry more information than the average across a pattern
Why Multi-Voxel Pattern Analysis (MVPA)?
Standard fMRI Analysis MVPA fMRI Analysis
Average Pattern
Questions?
• We have seen that the pattern of activity might be informative but it is lost in the standard fMRI analysis
How Multi-Voxel Pattern Analysis?
1. MVPA classifier
2. MVPA correlation
3. Representational similarity approach
TrainingTrials
TestTrials
(not in training set)
trial 1
Can an algorithm correctly “guess” trial identity better than
chance (50%)?
trial 3
trial 2
trial 1
trial 2
trial 3
… …
FACES HOUSES
Activ
ity in
Vox
el 1
Activity in Voxel 2FacesHouses
Each dot is one measurement (trial) from one condition (red circles) or the other (green circles)
Vox
el 1
Vox
el 2
Activ
ity in
Vox
el 1
Activity in Voxel 2
Classifier
Training set Test set
FacesHouses
Activ
ity in
Vox
el 1
Activity in Voxel 2
Classifier
Test set
Classifier Accuracy =
Correct
Incorrect 8
6= = 75 %FacesHouses
THIS STEP IS THE CORE OF THE ANALYSIS BECAUSE IT TESTS THE ABILITY OF THE CLASSIFIER TO GENERALIZE KNOWLEDGE
TO NEW DATA
Cox and Savoy (2003)
In a, the classifier can operate on single voxels because the response distributions are separable within individual voxels.
Faces
Houses
Cox and Savoy (2003)
Faces
Houses
Also standard fMRI analysis can detect this difference
Cox and Savoy (2003)
In b, the classifier cannot operate on single voxels because the response distributions are overlapping within individual voxels.
Faces
Houses
Cox and Savoy (2003)
Faces
Houses
Standard fMRI analysis cannot detect this difference
Cox and Savoy (2003)
In c, the non-linear classifier draws decision boundaries other than straight lines
Support Vector Machine (SVM)
Mur et al., 2009
Mur et al., 2009
• SVM is a linear decision boundary that discriminates between two sets of points with the constrain of a greater distance from the closest points on both sides.
• The response patterns closest to the decision boundary (yellow circles) that defined the margins are called “support vectors”.
Choices to makeFeature selection to remove noisy and/or uninformative voxels before classification:1.Limit the analysis to specific anatomical regions2.Compute univariate (voxel-wise) statistics
Cross-validation options:1.Leave-one-trial out: for each condition train the classifier with all but one trial and use the left out subset as test data. Repeat this procedure until each subset has been used as test data once.2.Leave-one-run out: train the classifier with all but one runs and test on the excluded run. Repeat until all runs are used as test data once.
Performance evaluation options:1.T-test versus 50% chance decoding2.Permutation test: train and test the classifier with labels (e.g. faces and houses) randomly assigned to the data. Compare classifier performance on real data with performance on randomized data.
Example of MVPA classifier approach: decoding future actions
Gallivan et al., 2011
PREDICT FUTURE EFFECTOR
Gallivan et al., 2011
PREDICT THE UPCOMING REACH OR
SACCADE
Gallivan et al., 2011
PREDICT FUTURE DIRECTION
PREDICT THE UPCOMING MOVEMENT DIRECTION
What else can we examine?
So far we’ve only examined the question:
Can we discriminate Condition 1 vs. Condition 2 from brain activation patterns?
New question:
Are different stimuli similarly represented in certain ROIs?
Answer:
Cross-decoding: train to discriminate between Condition 1 and 2 and test classifier accuracy with Condition 3 and 4
Cross-trial-type decoding
Activ
ity in
Vox
el 1
Activity in Voxel 2 FacesHousesClassifier
Training set
Cross-trial-type decoding
Test set
Activ
ity in
Vox
el 1
Activity in Voxel 2
ToolsAnimals
Classifier
Gallivan et al., 2011
Train to discriminate movement directions for one effector (HandL vs HandR)
and test it on the other effector (EyeL vs EyeR)
Similar spatial encoding across the eye and the hand
Effector specificity across different movement directions
Gallivan et al., 2011
Train to discriminate the effector for one movement directions (EyeL vs HandL)
and test it on the other direction (EyeR vs HandR)
Gallivan et al., 2011
Effector specificity across different movement direction
Questions?
• We have seen how MVPA classifier can be used to distinguish between different stimulus categories based on the pattern of activity
trial 1
trial 3
trial 2
trial 3
Faces Houses
AverageSummedActivation
MVPA correlation approach
trial1
trial2
trial3
trial1
trial2
trial3
trial 1
trial 3
trial 2
trial 3
Faces Houses
AverageSummedActivation
MVPA correlation approach
trial1
trial2
trial3
trial1
trial2
trial3
The same category evokes similar patterns of activity across trials
trial1
trial2
trial3
trial 1
trial 3
trial 2
trial 3
Faces Houses
AverageSummedActivation
MVPA correlation approach
trial1
trial2
trial3
Similarity Within the same category
trial1
trial2
trial3
trial 1
trial 3
trial 2
trial 1
trial 2
trial 3
Faces Houses
AverageSummedActivation
MVPA correlation approach
trial1
trial2
trial3
Similarity Between different categories
trial1
trial2
trial3
Within-category similarity Between-category similarity>
The brain area contains distinct information about faces and houses
Haxby et al., 2001, Science
Category-specificity of patterns of response in the ventral temporal cortex
Haxby et al., 2001, Science
Category-specificity of patterns of response in the ventral temporal cortex
SIMILARITY MATRIX
ODD RUNS
EV
EN
RU
NS
high
low
similarity
Within-category similarity
Haxby et al., 2001, Science
Category-specificity of patterns of response in the ventral temporal cortex
ODD RUNS
EV
EN
RU
NS
SIMILARITY MATRIX
high
low
similarity
Between-category similarity
Bracci et al., 2011
Similar representation for tools and hand not identified from the
standard univariate analysisfMR
I a
ctiv
ity
high
low
similarity
Searchlight analysis
We have seen how to search for patterns of information within a region of interest, but what if we want to know
“Where in the brain does the activity pattern contain information about the experimental condition?”
•Scanned the brain with a spherical multivariate “searchlight” centered on each voxel in turn (optimal searchlight has radius of 4 mm, that contains 33 2-mm-isovoxel voxels)
•Compute multivariate effect statistic at each location
Kriegeskorte et al., 2006
Questions?
• We have seen how MVPA correlation can measure similarities and differences between patterns of activity across conditions
PRE-DEFINED CATEGORIES
CATEGORY A CATEGORY B
Candies in different boxes are different because they have different prices
Which dimension to choose for the classification?
HOW ARE THESE OBJECTS CLASSIFIED IN THE BRAIN (BY PRIC€, COLOR, SHAPE, FLAVOURFLAVOUR OR OTHER DIMENSIONS)?
Representational similarity approach (RSA)
ODD RUNS
Kriegeskorte et al (2008)
EV
EN
RU
NS
MVPA correlation
• Differently from the MVPA correlation, RSA does not separate stimuli into a-priori categories
RSA
high
low
similarity
.
.
.
.
TRIA
LS
.
.
.
.
CON
DIT
ION
S
C1 CONDITIONS
C1
C96
No class boundaries!
high
low
similarity
C96
TEST VARIOUS PREDICTIONS
REALDATA
Kriegeskorte et al (2008)
high
low
similarity
Which prediction matrix is more similar to the real data?
Activation- vs. information-bases analysis
Kriegeskorte, Goebel & Bandettini, 2006
Activation-based (standard fMRI analysis):regions more strongly active during face than house
perception
Information-based (searchlight MVPA analysis):regions whose activity pattern distinguished the two
categories
35 % of voxels are marked only in the information-based map: category information is lost when data
are smoothed
Activation- vs. information-based analysis
Mur et al., 2009, Social Cognitive and Affective Neuroscience
Limitations of MVPA
• Good classification indicates presence of information (no necessarily neuronal selectivity) (Logothetis, 2008). E.g. successful face decoding in primary visual cortex
• Pattern-classifier analysis requires many decisions that affect the results (see Misaki et al., 2010)
Questions?
• We have seen how RSA can be used for exploratory analysis with complex stimulus set
• To summarize, the three MVPA methods are powerful especially to study distributed representations
Reconstruct new images
Miyawaki et al., 2008
• The model is trained to classify the pattern on brain activity in primary visual cortex during observation of several hundred random images
• The model can accurately reconstruct arbitrary images, including geometric shapes on a single 2 sec volume without any prior information about the image
Reconstruct new movies
(Nishimoto et al., 2011
• 3 participants watched hours of movie trailers during fMRI• Reconstruction of new movies from brain activity during similar clips
from YouTube
http://www.youtube.com/watch?v=nsjDnYxJ0bo
Presented movies
Reconstructed movies
Lie detector
(Davatzikos et al., 2005)
• Non-linear classifier applied to fMRI data to discriminate spatial patterns of activity associated to lie and truth in 22 individual participants.
• 88% accuracy to detect lies in participants not included in the training
Lie detector
• Non-linear classifier applied to fMRI data to discriminate spatial patterns of activity associated to lie and truth in 22 individual participants.
• 88% accuracy to detect lies in participants not included in the training
The real world is more complex!
Reconstruct dreams
• Measure brain activity while 3 participants were asleep and ask them to describe their dream when awake
• Comparison between brain activity during sleep and vision of pictures of categories frequently dreamt
• Activity in higher order visual areas (i.e. FFA) could successfully (accuracy of 75-80%) decode the dream contents 9 seconds before waking the participant!
Horikawa, Tamaki, Miyawaki and Kamitani, Society for Neuroscience 2012
http://www.youtube.com/watch?feature=player_embedded&v=u9nMfaWqkVE
Thousand of categories in the real world
• Which of the thousand of categories are represented similarly in the brain and which aren’t?
• Measure brain activity from 5 subjects while watching movies• See example movies:
Huth et al., 2012
Shared Semantic Space from brain activity during observation of movies
Similar colors for categories similarly represented in the brain
Huth et al., 2012
Similar colors for categories similarly represented in the brain
Shared Semantic Space from brain activity during observation of movies
People and communication verbs are represented similarly
Continuous Semantic Space across the surface
Each voxel is colored accordingly to which part of the semantic space is selective for
http://gallantlab.org/semanticmovies/
http://gallantlab.org/semanticmovies/
FUSIFORM FACE AREA
Click on each voxel to see which categories it represents
Continuous Semantic Space across the surface
Continuous Semantic Space across the surface
http://gallantlab.org/semanticmovies/
Click on each category to see how it is represented in the brain
FACE
The possibility to decode brain activity has great potentials for brain-computer applications
Let’s watch a video!
http://www.youtube.com/watch?v=TJJPbpHoPWo
