NeuroImage 96 (2014) 5466

Contents lists available at ScienceDirect

NeuroImage

j ourna l homepage: www.e lsev ie r .com/ locate /yn img

Multiclass fMRI data decoding and visualization using supervisedself-organizing maps

Lars Hausfeld , Giancarlo Valente, Elia FormisanoDepartment of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The NetherlandsMaastricht Brain Imaging Center, Faculty of Psychology and Neuroscience, Maastricht University, The Netherlands

Corresponding author at: Department of CognitiPsychology and Neuroscience, Maastricht UniversMaastricht, The Netherlands.

E-mail address: lars.hausfeld@maastrichtuniveristy.nl

http://dx.doi.org/10.1016/j.neuroimage.2014.02.0061053-8119/ 2014 Published by Elsevier Inc.

a b s t r a c t

a r t i c l e i n f o

Article history:Accepted 3 February 2014Available online 12 February 2014

Keywords:fMRIDecodingMulticlass classificationSelf-organizing maps

Whenmultivariate pattern decoding is applied to fMRI studies entailingmore than two experimental conditions,a most common approach is to transform the multiclass classification problem into a series of binary problems.Furthermore, for decoding analyses, classification accuracy is often the only outcome reported although the topol-ogy of activation patterns in the high-dimensional features spacemay provide additional insights into underlyingbrain representations. Herewe propose to decode and visualize voxel patterns of fMRI datasets consisting ofmul-tiple conditions with a supervised variant of self-organizing maps (SSOMs). Using simulations and real fMRI data,we evaluated the performance of our SSOM-based approach. Specifically, the analysis of simulated fMRI datawithvarying signal-to-noise and contrast-to-noise ratio suggested that SSOMs perform better than a k-nearest-neigh-bor classifier for medium and large numbers of features (i.e. 250 to 1000 or more voxels) and similar to supportvectormachines (SVMs) for small andmediumnumbers of features (i.e. 100 to 600 voxels). However, for a largernumber of features (N800 voxels), SSOMs performed worse than SVMs. When applied to a challenging 3-classfMRI classification problem with datasets collected to examine the neural representation of three human voicesat individual speaker level, the SSOM-based algorithmwas able to decode speaker identity from auditory corticalactivation patterns. Classification performances were similar between SSOMs and other decoding algorithms;however, the ability to visualize decodingmodels and underlying data topology of SSOMs promotes a more com-prehensive understanding of classification outcomes. We further illustrated this visualization ability of SSOMswith a re-analysis of a dataset examining the representation of visual categories in the ventral visual cortex(Haxby et al., 2001). This analysis showed that SSOMs could retrieve and visualize topography and neighborhoodrelations of the brain representation of eight visual categories.We conclude that SSOMs are particularly suited fordecoding datasets consisting of more than two classes and are optimally combined with approaches that reducethe number of voxels used for classification (e.g. region-of-interest or searchlight approaches).

2014 Published by Elsevier Inc.

Introduction

During the past years, the traditional voxel-wise analysis of fMRIdata has been complemented by so-called Multi-Voxel Pattern Analysis(MVPA; e.g. Haynes and Rees, 2006; Norman et al., 2006). In contrast tothe conventional activation-based statistical analysis, which examineseach voxel separately for activation differences between experimen-tal conditions using, for example, the General Linear Model (GLM;Friston, 1995), MVPA represents an information-based analysis(Kriegeskorte et al., 2006) that exploits information contained inspatial (i.e. multi-voxel) activation patterns.

Various decoding algorithms have been proposed in MVPA fMRIstudies. These algorithms differ in terms of their complexity and include

ve Neuroscience, Faculty ofity. P.O. Box 616, 6200 MD

(L. Hausfeld).

correlation-based approaches (Haxby et al., 2001), Gaussian-naveBayes classifiers (Mitchell et al., 2004), linear discriminant analysis(Cox and Savoy, 2003; Kriegeskorte et al., 2006), linear and non-linearsupport vector machines (SVMs; Cox and Savoy, 2003; LaConte et al.,2005; Mouro-Miranda et al., 2005) and sparse logistic regression(SLR; Miyawaki et al., 2008; Ryali et al., 2010; Yamashita et al., 2008).More recently, the combination of classification algorithms and voxelselection strategies (e.g. De Martino et al., 2008; Langs et al., 2011;Yamashita et al., 2008) and use ofmultiple classifiers has been proposed(e.g. Kuncheva and Rodrguez, 2010).

In this paper, we present an fMRI decoding approach based uponself-organizing maps (SOMs). SOMs were developed to visualize high-dimensional data by converting the topology of data points into simplegeometrical relationships on a two-dimensional grid (Kohonen, 2001).By preserving only themost important topological relationships, this al-gorithm abstracts from high-dimensional input and provides insightinto the underlying data structure. These properties rendered it an im-portant tool for data exploration in various domains. Supervised SOMs

http://crossmark.crossref.org/dialog/?doi=10.1016/j.neuroimage.2014.02.006&domain=pdfhttp://dx.doi.org/10.1016/j.neuroimage.2014.02.006mailto:lars.hausfeld@maastrichtuniveristy.nlhttp://dx.doi.org/10.1016/j.neuroimage.2014.02.006http://www.sciencedirect.com/science/journal/10538119

55L. Hausfeld et al. / NeuroImage 96 (2014) 5466

(SSOMs) inherit these characteristics and extend the SOMs suchthat they can classify unseen samples. The decoding of fMRI datawith SSOMs differs with respect to other approaches in two relevantaspects.

First, SSOMs are not restricted to binary comparisons and thus allowfor inherent multiclass decoding (i.e. when the fMRI measurements in-cludemore than two conditions). Many studies that analyzed fMRI databymeans ofMVPA employed experimental designswith two conditionsor translated multiclass problems into a series of binary comparisons(one-versus-one or one-versus-all schema) paired with post-hocprocessing (e.g. majority voting) to determine the predicted class(e.g. Beauchamp et al., 2009; Cox and Savoy, 2003; Ethofer et al.,2009; Kamitani and Tong, 2005; Mouro-Miranda et al., 2006;Reddy et al., 2010; Swisher et al., 2010; Walther et al., 2009). Onereason is that one of the most applied classification algorithms inMVPA fMRI applications, the SVM, is in its basic form restricted totwo class problems (but see Crammer and Singer (2002) andWeston and Watkins (1999) for multiclass SVM formulations andMartnez-Ramn et al. (2006) for decoding of fMRI data usingmulticlass SVM). So far, only few fMRI studies applied MVPA in amulticlass setting without employing binary comparisons and nec-essary post-processing. In these studies decoding was performedwith nave Bayes (Brouwer and Heeger, 2009; Mitchell et al., 2004)or neural network classifiers (Hanson et al., 2004; Polyn et al., 2005).

Second, SSOMs offer the possibility to visualize the underlyingdistri-bution of fMRI activity patterns used to train the decodingmodel. Thesevisualizations can reveal the topology of the underlying activity patternsin high-dimensional feature (voxel) space.When analyzing fMRI activa-tion patterns by MVPA, the experimenter is interested in the degree ofseparation between classes in feature space. In the case of two classes,the interpretation of classification results is straightforward, i.e. the ac-curacy of predictions describes the degree of separation of the responsepatterns. However, for classification of response patterns with morethan two classes, focusing on overall classification accuracy or perfor-mances of many binary comparisons provides only a partial, non-intuitive view on class distributions in high-dimensional voxel space.Thus, when performing multiclass classification an additional post-processing step is usually required to infer and visualize underlyingclass distributions (or topology), e.g. via confusion matrices. For exam-ple, Abdi et al. (2009) visualized class distributions in a multiclasssetting by first computing pairwise comparisons between classes andvisualizing, based on these outcomes, the underlying topology usingPrincipal Component Analysis (PCA) and a bootstrapped Multidimen-sional Scaling (MDS). It should be stressed that, in these cases, class dis-tributions are deduced using the classification performance of thedecoding algorithm. Conversely, SSOMs visualize the data directly andthus do not require additional post processing. Another approach thatallows visualizing data in fMRI is representational similarity analysis(RSA; Kriegeskorte et al., 2008a, 2008b). In RSA, representationaldissimilarity matrices (RDMs) are calculated using pairwise distances(e.g. correlation distance) between activity patterns evoked by distinctstimulus classes. Subsequently, topology of activity patterns can bevisualized by MDS of the dissimilarity matrices. A notable distinctioncompared to SSOMs is that RDMs are calculated from the entire featureset with equal weighting, whereas SSOMs-based topologies rely on theoptimized combination of features, which results from learning of thestimulus labels. Thus, SSOM-based visualization may lead to a betterabstraction of high-dimensional activity patterns.

Below we illustrate how SSOMs can be used in the context of fMRIdecoding to performmulticlass classification and visualize class topolo-gy. We present results on simulated fMRI data to illustrate the validityand usefulness of SSOM classification and visualization in the decodingof single data sets as well as in analyses involving multiple cross-valid