A Similarity Retrieval System for Multimodal Functional Brain Images
description
Transcript of A Similarity Retrieval System for Multimodal Functional Brain Images
![Page 1: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/1.jpg)
A Similarity Retrieval System for Multimodal Functional Brain
ImagesRosalia F. Tungaraza
Advisor: Prof. Linda G. Shapiro
Ph.D. Defense
Computer Science & EngineeringUniversity of Washington
1
![Page 2: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/2.jpg)
Functional Brain Imaging
Study how the brain works Imaging while subject performs a task Image represents some aspect of the brain e.g.
fMRI: brain blood oxygen level ERP: scalp electric activity
2
![Page 3: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/3.jpg)
Motivation
3
Given a database of functional brain images from various subjects, cognitive tasks, and image modality.
Database users need to retrieve similar images
A system that can automatically perform this retrieval will reduce amount of time and effort users spend during this task
3
![Page 4: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/4.jpg)
Content-Based Image Retrieval
4
• Given a query image and an image database, retrieve the images that are most similar to the query in order of similarity.
• Example system for photographic images: Andy Berman’s FIDS system; Yi Li’s Demo
http://www.cs.washington.edu/research/imagedatabase/demo
![Page 5: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/5.jpg)
Image Features / Distance Measures
Image Database
Query Image
Distance Measure
Retrieved Images
Image FeatureExtraction
User
Feature SpaceImages 5
![Page 6: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/6.jpg)
Contributions1. Created a similarity retrieval system for multimodal
brain images
I. fMRI, ERP, and combined fMRI-ERP
II. User interface
2. Developed feature extraction methods for fMRI and ERP data
3. Developed pair-wise similarity metrics
4. Simulated human expert similarity scores
6
![Page 7: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/7.jpg)
Outline Background
fMRI ERP Existing Similarity Retrieval Systems for
these modalities Feature Extraction Process Similarity Metric User Interface Retrieval Performance Simulate Human Expert
77
![Page 8: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/8.jpg)
Functional Magnetic Resonance Imaging (fMRI)
8
A non-invasive brain imaging technique Records blood oxygen level in brain While imaging, subject performs a task
8
![Page 9: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/9.jpg)
fMRI Statistical Images
Statistical Analysis
Voxel Thresholding
9
![Page 10: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/10.jpg)
Event-Related Potentials (ERP)
10
A non-invasive brain imaging technique Records electric activity along scalp While imaging, subject performs a task
@ 2004 by Nucleus Communications, Inc.
10
![Page 11: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/11.jpg)
ERP Source Localization
11
Researchers want to identify the electric activity and its source for each electrode
But, multiple sources for each electrode
LORETA approximates anatomic locations of sources
11
![Page 12: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/12.jpg)
Comparison of fMRI and ERP Data
12
fMRI ERP
Spatial resolution Good (in mm) undefined/poor
Temporal resolution Poor (in sec) Excellent (in msec)
12
![Page 13: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/13.jpg)
Similarity Retrieval Systems for fMRI Images
1313
![Page 14: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/14.jpg)
Similarity Retrieval Systems for ERP Images
14
No relevant literature found
14
![Page 15: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/15.jpg)
Similarity Retrieval Systems for Combined fMRI-ERP Images
15
No relevant literature found
15
![Page 16: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/16.jpg)
Outline Background
Feature Extraction Process fMRI features ERP features
Similarity Metric User Interface Retrieval Performance Simulate Human Expert
1616
![Page 17: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/17.jpg)
Threshold
Perform connected component analysis
Perform clustering
Approximate cluster centroids
Compute region vectors
Original database
fMRI Feature Extraction
k=3
k=2
k=1
1. Centroid 2. Avg Activation Value 3. Var Activation Value4. Volume5. Avg Distance to
Centroid6. Var of those Distances
17
![Page 18: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/18.jpg)
Select time-segment
Compute voxel-wise statistically significant difference between means
Threshold
ERP Feature Extraction
Compute feature(X,Y,Z)
positions of retained voxels
18
Signals at each point incorporateinformation from that voxel and neighbors.
The retained voxels have significant activationmeaning activities A and B are very different.
![Page 19: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/19.jpg)
Outline Background Feature Extraction Process
Similarity Metric Summed Minimum Distance Similarity Score for Combined fMRI-ERP
Images
User Interface Retrieval Performance Simulate Human Expert
1919
![Page 20: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/20.jpg)
Summed Minimum Distance (SMD) for fMRI and ERP Images
20
Subject Q Subject T
Q2T =
SMD = (Q2T+T2Q) / 220
Euclideandistancebetweenfeaturevectors*
*We also used normalized Euclidean distance.
![Page 21: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/21.jpg)
Sample SMD Scores
21
SMD
21
![Page 22: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/22.jpg)
Similarity Score for Combined fMRI-ERP Images
SIM(i,j) = αSMDfMRI(i,j) + (1-α)SMDERP(i,j)
22
![Page 23: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/23.jpg)
Outline
Background Feature Extraction Process Similarity Metric User Interface
Retrieval Performance Simulate Human Expert
2323
![Page 24: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/24.jpg)
GUI: Front Page
24
![Page 25: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/25.jpg)
GUI: Retrievals with SMD Scores
25
SMD
25
![Page 26: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/26.jpg)
GUI: Query-Target Activations (fMRI)
26
![Page 27: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/27.jpg)
GUI: Query-Target Activations (ERP)
27
![Page 28: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/28.jpg)
Outline Background Feature Extraction Process Similarity Metric User Interface
Retrieval Performance Data Sets fMRI Retrieval Performance ERP Retrieval Performance Combined fMRI-ERP Retrieval Performance
Simulate Human Expert 2828
![Page 29: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/29.jpg)
Data Sets for fMRI Retrievals
29
Central-Cross -- 24 subjects (Face Recognition)
AOD -- 15 subjects (Sound Recognition)
SB -- 15 subjects (Memorization)
Checkerboard -- 12 subjects (Face Recognition)
d g f p g n
29
![Page 30: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/30.jpg)
Data Set for ERP Retrievals
30
View Human Faces (Face Up) -- 15 subjects
View Houses (House Up) -- 15 subjects
… …
30
![Page 31: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/31.jpg)
Data Set for Combined fMRI-ERP Retrievals
31
ERP: same data set as used in ERP retrieval
fMRI: • Task: Face recognition using a house up
background
• Same subjects and images as data set for ERP retrieval
31
![Page 32: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/32.jpg)
fMRI Retrieval Performance
32
1. RFX Retrievals
2. Individual Brain Retrieval
3. Testing Group Homogeneity
4. Feature Selection
32
Random effects models are very conservativeaverage activation models from a group, whichcontain only activated voxels present in all members.
![Page 33: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/33.jpg)
fMRI Retrieval Score
33
Perfect score : Retrieval Score = 0
Random score: Retrieval Score ~ 0.5
Worst score: Retrieval Score = 1
33
![Page 34: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/34.jpg)
Example Scores
34
• Let N = 100 and Nrel = 3
• Sample Case1 Ri = i, i = 1 to 3 1 + 2 + 3 – 6 = 0/300
• Sample Case 2 R1 = 3, R2 = 2, R3 = 1 3 + 2 + 1 – 6 = 0/300
• Sample Case 3: R1 = 10, R2 = 20, R3 = 30 10 + 20 + 30 – 6 = 54/300
![Page 35: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/35.jpg)
fMRI Individual Brain Retrievals
35
Use individual brain as query
Mean Retrieval Scores (Top 6% activated voxels)
Checkerboard 0.09
SB 0.16
Central-Cross 0.21AOD 0.26
35
![Page 36: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/36.jpg)
Testing Group Homogeneity for fMRI
36
CB
AOD
Central-Cross
SB
36
MDS1 and MDS2 are 2 projections of themultidimensional feature data.
All groups except AOD had tight clusters.
![Page 37: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/37.jpg)
ERP Retrieval Performance
3737
![Page 38: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/38.jpg)
Subject #8 RetrievalsTo
p Re
trie
vals
Botto
m
Retr
ieva
ls
38
![Page 39: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/39.jpg)
Combined fMRI-ERP Retrieval
SIM(i,j) = αSMDfMRI(i,j) + (1-α)SMDERP(i,j)
α = 0.0, ERP only
α = 1.0, fMRI onlyα = 0.6
α = 0.3
39
![Page 40: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/40.jpg)
Outline Background Feature Extraction Process Similarity Metric User Interface Retrieval Performance
Simulate Human Expert Simulation Method Data Set Testing Function Performance
4040
![Page 41: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/41.jpg)
Simulate Human Expert Current retrieval system requires some expert
knowledge
Estimate a function to generate similarity scores with high correlation to expert scores
41
Dr. JeffOjemann
![Page 42: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/42.jpg)
Simulation Method1. Uniform feature representation: create
codebook and encode each subject
2. Concatenate the codebook features for each pair of subjects
3. Create eigenfeatures
4. Estimate a function
5. Test function performance42
![Page 43: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/43.jpg)
The Codebook
43
• Out of all the clusters found in all N brains, create a single brain that has a representation of each unique cluster. This is the codebook.
• Then for each of the N brains use the codebook to create a subject-specific vector representing each of those clusters.
• In the case where the codebook has a given cluster, but that particular subject misses it, that whole portion of this subject's codebook will be empty.
• Otherwise, the other parts of this subject's codebook will be filled with the properties of this subject's clusters.
![Page 44: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/44.jpg)
S1 S2 S3
S1 S2 S3
Create the reference brain (codebook)
Encode Images
1. Uniform Feature Representation
44
![Page 45: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/45.jpg)
2. Concatenate Codebook Features
XYZ Centroid A Avg Activation Value VA Var Activation ValueS Size (Volume)D Avg Distance to CentroidVD Var of those Distances
45
Pairs of Subjects Expert Score
![Page 46: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/46.jpg)
3. Create Eigenfeatures
46
originalfeaturespace
eigenfeaturespace
![Page 47: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/47.jpg)
4. Estimate a Function
Linear function using linear regression Non-linear function using generalized
regression neural networks (GRNN)
S1,S1
S1,S2
S1,S3
S3,S3
… … … … … … … …
0.00
0.30
0.90
0.00
…
47
We want to estimate a function that takes a pair of region vectorsfrom two subjects and computes their similarity score.
![Page 48: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/48.jpg)
5. Test Function Performance The Pearson Correlation Coefficient (CC)
The Average Absolute Error (A-ABSE)
The Root Mean Square Error (RMSE)
48
![Page 49: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/49.jpg)
Data Set
fMRI data (Central-Cross)
-- 23 subjects
-- Face Recognition task
+Human Expert Generated Pair-wise Similarity Matrix
49
![Page 50: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/50.jpg)
Overall Function Performance
50
overfitting!
![Page 51: A Similarity Retrieval System for Multimodal Functional Brain Images](https://reader035.fdocuments.in/reader035/viewer/2022062323/5681632e550346895dd3aa35/html5/thumbnails/51.jpg)
Contributions1. Created a similarity retrieval system for multimodal
brain images
I. fMRI, ERP, and combined fMRI-ERP
II. User interface
2. Developed feature extraction methods for fMRI and ERP data
3. Developed pair-wise similarity metrics
4. Simulated human expert similarity scores
51