Classifying Instantaneous Cognitive States from fMRI Data
-
Upload
harrison-bradley -
Category
Documents
-
view
18 -
download
1
description
Transcript of Classifying Instantaneous Cognitive States from fMRI Data
![Page 1: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/1.jpg)
1
Classifying Instantaneous Cognitive States from fMRI Data
Tom Mitchell, Rebecca Hutchinson, Marcel Just, Stefan Niculescu, Francisco Pereira, Xuerui Wang
Carnegie Mellon University
November, 2003
![Page 2: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/2.jpg)
2
…
![Page 3: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/3.jpg)
3
![Page 4: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/4.jpg)
4
Cognitive state sequence
COGNITIVE TASK
“Virtual sensors” of cognitive state
1. Does fMRI contain enough information?
2. Can we devise learning algorithms to construct such “virtual sensors”?
![Page 5: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/5.jpg)
5
Learning Virtual Sensors
• Learn fMRI(t,t+k) CognitiveState
• Classifiers:– Gaussian Naïve Bayes, SVM, kNN
• Trained per subject, per experiment
• Feature selection/abstraction– Select subset of voxels (by signal, by anatomy)– Select subinterval of time– Average activities over space, time– Normalize voxel activities
![Page 6: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/6.jpg)
6
Study 1: Pictures and Sentences
• Trial: read sentence, view picture, answer whether
sentence describes picture
• Picture presented first in half of trials, sentence first
in other half
• Three possible objects: star, dollar, plus
• Collected by Just et al.
![Page 7: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/7.jpg)
7
It is true that the star is above the plus?
![Page 8: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/8.jpg)
8
![Page 9: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/9.jpg)
9
+
---
*
![Page 10: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/10.jpg)
10
![Page 11: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/11.jpg)
11
Is Subject Viewing Picture or Sentence?
• Learn fMRI(t,t+8) {Picture, Sentence}
• Leave two out cross-validation was used to assess the performance of the classifiers
• SVMs and GNB worked better than kNN
• Some Details: – 12 subjects, 40 pictures, 40 sentences– 1397 - 2864 voxels per subject, 7 ROIs – fMRI snapshot taken every half second
![Page 12: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/12.jpg)
12
Error for Single-Subject Classifiers
•Error computed by averaging over all subjects
•95% confidence intervals per subject are ~ 10% large
• Error of default classifier is 50%
Dataset \ Classifier GNB SVM 1NN 2NN 5NN
Picture vs Sentence 0.16 0.09 0.20 0.18 0.19
![Page 13: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/13.jpg)
13
• Approach: define supervoxels based on anatomically defined regions of interest– Normalize per voxel activity for each subject
• Each value scaled now in [0,1]
– Abstract to seven brain region supervoxels– 16 snapshots for each supervoxel
• Train on n-1 subjects, test on nth– Leave one subject out cross validation
Can We Train Subject-Indep Classifiers?
![Page 14: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/14.jpg)
14
• NO Feature Selection used in this experiment
•95% confidence intervals approximately 5% large
•Error of default classifier is 50%
Error for Cross Subject Classifiers
Dataset \ Classifier GNB SVM 1NN 2NN 5NN
Cross-Subject Pict vs Sent
0.30 0.25 0.36 0.33 0.32
![Page 15: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/15.jpg)
15
Study 2: Word Categories
• Family members
• Occupations
• Tools
• Kitchen items
• Dwellings
• Building parts
• 4 legged animals
• Fish
• Trees
• Flowers
• Fruits
• Vegetables
![Page 16: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/16.jpg)
16
Word Categories Study
• Stimulus:
– 12 blocks of words:
• Category name (2 sec)
• Word (400 msec), Blank screen (1200 msec); answer
• Word (400 msec), Blank screen (1200 msec); answer
• …
– Subject answers whether each word in category
– 20 words per block, nearly all in category
![Page 17: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/17.jpg)
17
Training Classifier for Word Categories
• Learn fMRI(t) Word Category
• Training methods: kNN, GNB
• Leave one example out from each class used to assess performance
• Some Details: – 10 subjects, 20 examples per class– 8470 - 11,136 voxels per subject, 30 ROIs– fMRI snapshot taken every second
![Page 18: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/18.jpg)
18
Study 2: Results
Classifier outputs ranked list of classesEvaluate by the fraction of classes ranked ahead of true
class– 0=perfect, 0.5=random, 1.0 unbelievably poor
Dataset \ Classifier
GNB 1NN 3NN 5NN
Words 0.08 0.30 0.20 0.16
![Page 19: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/19.jpg)
19
Study 3: Syntactic Ambiguity
• Is subject reading ambiguous or unambiguous sentence?– “The experienced soldiers warned about the dangers
conducted the midnight raid.”
– “The experienced soldiers spoke about the dangers before the midnight raid.”
• Almost random results if no feature selection used • With feature selection:
– SVM - 77% accuracy
– GNB - 75% accuracy
– 5NN – 72% accuracy
![Page 20: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/20.jpg)
20
• Four feature selection methods:
• Active (n most active available voxels compared to baseline fixation activity, according to a t-test)
• RoiActive (n most active voxels in each ROI)
• RoiActiveAvg (average of the n most active voxels in each ROI)
• Disc (n most discriminating voxels according to a trained classifier)
• Active works best
Feature Selection
![Page 21: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/21.jpg)
21
Dataset
Feature
Selection
GNB SVM 1NN 3NN 5NN
PictureSent
No 0.29 0.32 0.43 0.41 0.37
Active 0.16 0.09 0.20 0.18 0.19
Words
No 0.10 N/A 0.40 0.40 0.40
Active 0.08 N/A 0.30 0.20 0.16
SyntAmb
No 0.43 0.38 0.50 0.46 0.47
Active 0.25 0.23 0.29 0.29 0.28
Feature Selection
![Page 22: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/22.jpg)
22
Summary
• Proved that there is enough information in the fMRI signal to allow decoding of Cognitive States
• Successful training of classifiers for instantaneous cognitive state in three studies
• Cross subject classifiers trained by abstracting to anatomically defined ROIs
• Feature selection and abstraction are essential
![Page 23: Classifying Instantaneous Cognitive States from fMRI Data](https://reader035.fdocuments.in/reader035/viewer/2022062718/56812e80550346895d942283/html5/thumbnails/23.jpg)
23
Research Opportunities
• Learning temporal models– HMM’s, Temporal Bayes Nets
• Learn to discriminate whether a subject has certain mental disease
• Discovering useful data abstractions– ICA, PCA, hidden layers in Neural Nets
• Merging data from multiple sources– fMRI, ERP, reaction times