Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive...
Transcript of Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive...
![Page 1: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/1.jpg)
Mesh Learning for Classifying
Cognitive Processes
Mete Özay*, Orhan Fırat*
İlke Öztekin**, Uygar Öztekin***,
Fatoş T. Yarman Vural*
* Middle East Technical University, Ankara
** Koc University, Istanbul
![Page 2: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/2.jpg)
Motivation:
Can we model the brain
activities measured by fMRI as
a machine learning system?
Algorithms
that can
learn
Human Machine
2
![Page 3: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/3.jpg)
Focus: Design a classifier to
model the distributed patterns of
activity in memory
Machine Human _
1.Training 1. Encoding: subject studies
objects from a category
2.Test 2. Retrieval: subject is
asked to recognize a test
object
3
![Page 4: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/4.jpg)
fMRI Data Acquisition
4
(Öztekin & McElree, 2007; Öztekin et al., 2009; Öztekin & Badre, 2011)
![Page 5: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/5.jpg)
Image samples
5
![Page 6: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/6.jpg)
fMRI dataset
10 semantic categories:
animals, colors, furniture, body parts, fruits,
herbs, clothes, chemical elements, vegetables
and tools.
Dataset:
24 samples /category
240 training + 240 test samples from the encoding and retrieval phase and
Number of voxels:
Memory: 8142
Whole Brain: 82 600
6
![Page 7: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/7.jpg)
fMRI intensity values of a voxel
7
![Page 8: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/8.jpg)
Spatial Distribution of Voxel
Intensities for a time instant t
8
![Page 9: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/9.jpg)
fMRI intensity values as a
function of time
9
State of the art
performance
~25-30% (e.g. Öztekin & Badre,
2011)
![Page 10: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/10.jpg)
Spatio-temporal Distribution of
Voxel Intensities
10
![Page 11: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/11.jpg)
Voxel intensity values for 10 class at 10
neighboring voxels
11
![Page 12: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/12.jpg)
Difference of intensity values
between two neighboring voxels
12
![Page 13: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/13.jpg)
Neurons are massively
interconnected Relationships among the voxels are more
discriminative then the individual voxel
intensity values to represent a certain
category
Need to model the relationships among the
voxels
13
![Page 14: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/14.jpg)
A Local Mesh Model:
1 , { , : , , }i j i k j k j l i lv t s v t s s s s s v t s D
1 1, { , , : , , , }c
p i j i k p i j j k j l i l p i jv t s v t s v t s s s s s v t s v t s
, , ,, ,k p
i j i j k i k i j
s
v t s a v t s
, ,, , ,p i j i j p i j kM v t s v t s a A
14
A voxel is represented in
a neighborhood system
![Page 15: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/15.jpg)
Local Relational Features: LRF
1, 2, , . . . T
j j j N jA a a a
ε2ij
, , ,1 , ,2 , , ...i j i j i j i j pa a a a
![Page 16: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/16.jpg)
Mesh Learning with Spatial
Neighborhood
16
![Page 17: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/17.jpg)
Performance of Mesh Learner with Spatial
Neighborhood 10 class classification performances using 8142 voxels.
17
ICA: Independent component analysis
PCA: Principal Component Analysis
KPCA: Kernel Principal Component Analysis
![Page 18: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/18.jpg)
Experiments on Mesh Learner with
Spatial Neighborhood
Single voxel performance for 10 classes. 18
![Page 19: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/19.jpg)
19
![Page 20: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/20.jpg)
20
![Page 21: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/21.jpg)
Discussion on Mesh Learning
with Spatial Neighborhoods Spatial neighborhood with
L2-norm implies
anatomical surroundings
of a voxel; which may not
be the case in cognitive
process.
Selecting the optimal
value of p is not validated
and introduced as a user
parameter.
Employ functional
connectivity.
Find voxels which are
highly correlated to other
voxel. p changes for each
voxel.
21
![Page 22: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/22.jpg)
Mesh Learning with Functional
Neighborhood
1. Need to define functional connectivity among
the voxels
2. Define Functional Neighborhood
3. Apply functional neighborhood to k-nn to and
select k-functionally-closest neighbors which
implies coupled-activation in cognitive
process
22
![Page 23: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/23.jpg)
Functional Connectivity
Statistical association or
dependency among the
time series of voxels
23
![Page 24: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/24.jpg)
Modeling Methods of Brain Using Connectivity
1. Find correlation between time series of two voxels, using a
correlation metric .
2. Construct correlation matrix by using correlation measure of
each pair of voxels. 24
![Page 25: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/25.jpg)
Cross Correlation Metric
Cross-correlation of any two individual time-
series (i,j), at lag h, ρij(h) , is defined as
25
![Page 26: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/26.jpg)
Scalability of Functional
Connections
Connectivity matrices are expensive in voxel
level, when no approximations are made
Considering functional relations of a voxel with
all other voxels;
8142 voxels makes 33M functional relations
26
![Page 27: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/27.jpg)
Design and Use of Functional
Connectivity Cluster voxels by their locations
Measure correlation metric within clusters to
generate connectivity matrices
Use connectivity matrices to find functionally-
nearest neighbors
27
![Page 28: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/28.jpg)
k=256 sub-regions
k=256 of each
sub-regions
s≅35 voxels
. ...
.
Functional
connectivity map
among the clusters
Functional connectivity for each cluster, for a given class
28
![Page 29: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/29.jpg)
Design of New Neighborhood System
Rather then selecting p-spatially closest points by
L2-norm; select p-functionally closest points
Select p-functionally closest points analyzing rows of
within-cluster connectivity matrix
Construct neighborhood-set with p-functionally
closest voxels and calculate LRF accordingly
29
![Page 30: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/30.jpg)
Given a voxel Select p-Functionally
Closest Voxel(s)
Selecting p=4 functionally
closest voxels vj
For voxel vi in cluster ck
where i=1 and k=26;
Resulting neighbor indexes by
considering highly correlated
voxels in the cluster:
j={3,6,9,14}
30
![Page 31: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/31.jpg)
Classification Performances (%) with Spatial and
Functional connectivity
% LRF Order Values with Functional Connectivity (p) Without FC (raw LRF)
2 3 4 5 6 7 8 9 10 8 9 10
k-nn 61,9384 62,3732 62,3551 64,4565 62,7899 64,0399 60,2717 61,9384 61,1051 56 56 57
![Page 32: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/32.jpg)
Performance of Mesh Learning with
Functional Connectivities, p=10
32
![Page 33: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/33.jpg)
0,6250
0,7500
0,7917
0,6957
0,5417
0,6250
0,6667
0,7083 0,7083
0,5417
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1 2 3 4 5 6 7 8 9 10
Perf
orm
an
ce (
%)
Class Label
Knn Classification for Each Class with Best FC Aware LRF
Her sınıf için elde edilen en yüksek performans
değeri
![Page 34: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/34.jpg)
Conclusion
Mesh Learning model
allows us to identify and differentiate classes
of information represented in the brain during
memory encoding and retrieval processes
Functional connectivity represents the mesh
better than the spatial connectivity
34
![Page 35: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/35.jpg)
Implications
We ultimately aim to read minds
Better understand intention
Better interpret feedback
...
Although we are not there yet, we are as
close as we can get!
35
![Page 36: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/36.jpg)
Thanks to Google
Project Website:
neuro.ceng.metu.edu.tr
36
![Page 37: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/37.jpg)
Open Issues
Estimating the true number of clusters
Hierarchical neighbor selection
Network measures will be incorporated
Combination and use of between cluster
metrics
37
![Page 38: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/38.jpg)
![Page 39: Mesh Learning for Classifying Cognitive Processes · Mesh Learning for Classifying Cognitive Processes Mete Özay*, Orhan Fırat* İlke Öztekin**, Uygar Öztekin***, Fatoş T. Yarman](https://reader034.fdocuments.in/reader034/viewer/2022050719/5f4f12994ccbea035e187651/html5/thumbnails/39.jpg)