The co-occurrence of multisensory facilitation and cross ......sounds and visual objects. Behavioral...

The co-occurrence of multisensory facilitation and cross-modal conflict in thehuman brain

Andreea Oliviana Diaconescu, Claude Alain, and Anthony Randal McIntosh1Rotman Research Institute, Baycrest Centre, Toronto; and 2Department of Psychology, University of Toronto, Ontario, Canada

Submitted 5 April 2011; accepted in final form 26 August 2011

Diaconescu AO, Alain C, McIntosh AR. The co-occurrence ofmultisensory facilitation and cross-modal conflict in the human brain.J Neurophysiol 106: 2896–2909, 2011. First published August 31,2011; doi:10.1152/jn.00303.2011.—Perceptual objects often com-prise a visual and auditory signature that arrives simultaneouslythrough distinct sensory channels, and cross-modal features are linkedby virtue of being attributed to a specific object. Continued exposureto cross-modal events sets up expectations about what a given objectmost likely “sounds” like, and vice versa, thereby facilitating objectdetection and recognition. The binding of familiar auditory and visualsignatures is referred to as semantic, multisensory integration.Whereas integration of semantically related cross-modal features isbehaviorally advantageous, situations of sensory dominance of onemodality at the expense of another impair performance. In the presentstudy, magnetoencephalography recordings of semantically relatedcross-modal and unimodal stimuli captured the spatiotemporal pat-terns underlying multisensory processing at multiple stages. At earlystages, 100 ms after stimulus onset, posterior parietal brain regionsresponded preferentially to cross-modal stimuli irrespective of taskinstructions or the degree of semantic relatedness between the audi-tory and visual components. As participants were required to classifycross-modal stimuli into semantic categories, activity in superiortemporal and posterior cingulate cortices increased between 200 and400 ms. As task instructions changed to incorporate cross-modalconflict, a process whereby auditory and visual components of cross-modal stimuli were compared to estimate their degree of congruence,multisensory processes were captured in parahippocampal, dorsome-dial, and orbitofrontal cortices 100 and 400 ms after stimulus onset.Our results suggest that multisensory facilitation is associated withposterior parietal activity as early as 100 ms after stimulus onset.However, as participants are required to evaluate cross-modal stimulibased on their semantic category or their degree of congruence,multisensory processes extend in cingulate, temporal, and prefrontalcortices.

multisensory integration; auditory; visual; magnetoencephalography;event-related synthetic aperture magnetometry analysis

PREVIOUS EXAMINATIONS OF MULTISENSORY processes demon-strated that perception by means of distinct sensory channelsnot only contributes to the richness of each sensory experiencebut also improves object detection and recognition (for areview, see Doehrmann and Naumer 2008). Simple low-levelstimulus properties, including temporal correspondence andspatial congruence, mediate multisensory facilitation in audi-tory and visual modalities. Indeed, a stimulus presented in onesensory modality can facilitate the detection of a spatiallycoincident stimulus presented in a distinct sensory modality.

McDonald et al. (2000) used an auditory cross-modal cueingparadigm to demonstrate that the co-occurrence of an irrelevantsound decreased the speed of detection of a subsequent lightwhen the target light appeared in the same location as thesound. In addition, Diederich and Colonius (2004) providedevidence of faster response times (RTs) to monochromaticlights when they were accompanied by simple tones presentedsimultaneously or at short intervals before or after targetstimuli. Therefore, a temporally synchronized stimulus in onesensory modality facilitates the detectability of another stimu-lus presented in another sensory modality.

Behavioral facilitation in multisensory contexts capitalizeson the spatial and temporal properties of cross-modal, audio-visual (AV) stimuli as well as their semantic relatedness. Thecritical stimuli used in the present study included complexsounds and semantically related black-and-white line drawingsof animate and inanimate objects. We examined the neuralprocesses that mediate the associations between familiar audi-tory and visual stimuli, which can be referred to as semanticmultisensory integration resembling naturalistic situations ofmultisensory stimulation. Multisensory integration, in the pres-ent context, refers to the neural process by which unisensorysignals are combined to form a unique signal that is specificallyassociated with the cross-modal stimulus. It is operationallydefined as a multisensory response, both neural and behavioral,that is significantly distinct from the sum of the responsesevoked by the modality-specific component stimuli (cf. Stein etal. 2010).

Whereas integration of semantically congruent cross-modalstimuli speeds up object detection and recognition, cross-modal conflicts impair performance (Chen and Spence 2010).Previous research demonstrated the presence of visual domi-nance during cross-modal conflicts, with ambiguous visualinputs impairing the recognition of complex sounds more thanambiguous auditory information impaired visual object recog-nition (Laurienti et al. 2004; Yuval-Greenberg and Deouell2009). Furthermore, studies from Spence and colleagues(Hartcher-O’Brien et al. 2008; Koppen et al. 2008; Koppen andSpence 2007a, 2007b; Sinnett et al. 2007, 2008) reportedlonger RTs following cross-modal stimuli when participantswere required to detect the presence of simple tones embeddedwithin cross-modal presentations. These findings further sup-port the interference of the more dominant visual sensorymodality on the detection of auditory targets.

To date, few studies have examined the effects of bothmultisensory facilitation and competition on performance. Sin-nett et al. (2008) showed that presentations of complex soundsalongside visual objects can either speed up or slow down RTsdepending on task demands. Participants were significantly

Address for reprint requests and other correspondence: A. O. Diaconescu,Laboratory for Social and Neural Systems Research, Dept. of Economics, Univ. ofZürich, Blümlisalpstrasse 10, CH-8006, Zürich, Switzerland (e-mail:[email protected]).

J Neurophysiol 106: 2896–2909, 2011.First published August 31, 2011; doi:10.1152/jn.00303.2011.

2896 0022-3077/11 Copyright © 2011 the American Physiological Society www.jn.org

by 10.220.32.247 on Septem

ber 29, 2016http://jn.physiology.org/

Dow

nloaded from

http://jn.physiology.org/

faster to detect visual targets when they were embedded withincross-modal presentations than when they were presentedalone. However, when required to identify complex sounds thatwere either presented in isolation or as cross-modal pairs,participants were significantly slower to respond to cross-modal stimuli compared with unimodal ones.

Given the behavioral advantages of multisensory integrationand the performance decreases following cross-modal con-flicts, more research is required to determine whether a com-mon network of brain regions is responsible for processingcross-modal conflict compared with multisensory facilitationprocesses. Furthermore, more research is required to determinethe extent to which multisensory processes are modulated bythe degree of semantic correspondence between visual andauditory stimuli. Thus, in the present study, we used magne-toencephalography (MEG) recordings to investigate the neuralmechanisms underlying both multisensory facilitation andcross-modal conflict during the presentation of complex soundsand semantically related black-and-white line drawings ofanimate and inanimate objects.

Noninvasive electrophysiological studies in humans thatused semantically related AV stimuli and contrasted cross-modal with unimodal stimulus presentations demonstratedmodulations of both early and late event-related related poten-tials (ERPs) over both sensory-specific and modality-indepen-dent cortical areas (e.g., Molholm et al. 2004; Yuval-Green-berg and Deouell 2007). For instance, recent studies foundincreases in the visual N1 waveform at occipital sites forcross-modal presentations compared with unimodal ones (Mol-holm et al. 2004; Yuval-Greenberg and Deouell 2007). Yuval-Greenberg and Deouell (2007) also examined oscillatory gam-ma-band activity (30–70 Hz) in response to cross-modal stim-uli using a name verification task of naturalistic objects. Earlyand late gamma-band activity at 90 and 260 ms, respectively,was associated with low-level feature integration and higherlevel object representation. Similarly, using a perceptual deci-sion task, Schneider et al. (2008) found enhanced gamma-bandactivity between 120 and 180 ms in response to semanticallycongruent relative to semantically incongruent cross-modalpairs of common objects.

There is increasing evidence that meaning and semanticrelatedness play an important role in multisensory integration.For instance, Senkowski et al. (2007) compared the effects ofnatural and abstract stimuli reflecting motion. Participants werepresented with a random stream of naturalistic and abstractvideo clips and static target stimuli. Each stimulus class con-sisted of unimodal auditory, unimodal visual, and cross-modalpresentations. The participants’ task was to count the targetstimuli embedded within the natural and abstract stimulusclasses. Only the former showed evidence of early multisen-sory integration within 120 ms after stimulus onset. Further-more, naturalistic motion represented by AV presentationsengaged occipital, temporal, and frontal regions.

Murray et al. (2004), on the other hand, demonstrated theefficacy of cross-modal stimuli on subsequent memory recallby showing that visual images were more likely to be remem-bered when they were previously presented along with seman-tically related complex sounds than when they were presentedonly in the visual modality. Cross-modal presentations, in thiscase, led to increases in cortical activity in the right lateral

occipital complex (LOC), which has been previously linked tovisual object recognition (Malach et al. 1995).

Electrophysiological research of semantic cross-modal con-flicts has focused primarily on conflicts regarding the phoneticperception of speech. Whereas congruent articulatory gesturessignificantly speed up auditory processes related to speech (vanWassenhove et al. 2005), incompatible visual input can greatlyimpair speech perception. The most documented case of cross-modal conflict during speech is the McGurk effect (McGurkand MacDonald 1976), in which incongruent or temporallyasynchronous visual representations of speech modify the au-ditory percept phonetically (McGurk and MacDonald 1976).

Several MEG studies (e.g., Kaiser et al. 2005; Nishitani andHari 2002) showed oscillatory gamma-band activity in re-sponse to McGurk-like stimuli beginning in the posteriorparietal scalp region at 160 ms and extending over occipitaland temporal-to-inferior frontal regions between 200 and 320ms. To determine the brain regions that are chiefly involved inmultisensory integration during speech, previous studies usedtranscranial magnetic stimulation (TMS) and temporally per-turbed task-related neural activity. Disruption of posteriorparietal activity 200 ms after stimulus onset impaired AVintegration of tones and shapes (Pourtois and de Gelder 2002).In addition, TMS applied to the superior temporal sulcus (STS)significantly reduced the McGurk effect within 100 ms afterauditory syllable onset (Beauchamp et al. 2010).

With respect to semantically incongruent combinations ofnonspeech sounds and visual objects, large negative amplitudedeflections were captured 390 ms after stimulus onset (Mol-holm et al. 2004). This ERP pattern resembled the N400component, which was previously implicated in semantic mis-match processing of objects in linguistic contexts (Kutas andFedermeier 2000). Although the effects of cross-modal incon-gruence were not explicitly examined, recent functional mag-netic resonance imaging (fMRI) studies suggested that cingu-late and inferior frontal cortices were recruited during semanticconflicts (Laurienti et al. 2003; Van Petten and Luka 2006).

In the present study, we investigated the effects of bothmultisensory facilitation and semantic cross-modal conflicts insensory-specific and modality-independent brain regions. Wepredicted that the integrated neural signal would be larger and,most importantly, that multisensory processes would engage adistinct set of brain regions than each of the responses evokedby the modality-specific component stimuli. We considered itto be a principled prediction that multisensory integrationprocesses are not simply the linear combination of unisensoryresponses. Recent multiple cell recording studies (Molholm etal. 2006) and functional neuroimaging studies in humans(Baumann and Greenlee 2007; Bishop and Miller 2008; Cal-vert et al. 2001; Grefkes et al. 2002; Macaluso et al. 2004;Moran et al. 2008) showed that cross-modal stimuli not onlyelicited increased activity in sensory-specific cortices but alsoactivated a distinct network of posterior parietal brain regions,including the inferior parietal sulcus (IPS), the inferior parietallobule (IPL), and the superior parietal lobule (SPL).

All conditions used in the present study included bothunimodal and cross-modal presentations of familiar complexsounds and visual objects. Behavioral multisensory facilitationeffects that capitalize on the semantic relatedness of cross-modal stimuli were assessed by asking participants to catego-rize stimuli based on their animacy (i.e., living vs. nonliving

2897MULTISENSORY FACILITATION AND CROSS-MODAL CONFLICT

J Neurophysiol • VOL 106 • DECEMBER 2011 • www.jn.org

by 10.220.32.247 on Septem


Dow

nloaded from


objects). Conversely, both behavioral and neural responses tocross-modal conflicts were examined by requiring participantsto detect the level of semantic correspondence between audi-tory and visual stimuli embedded within cross-modal presen-tations that were either semantically congruent or incongruent.We were particularly interested to compare multisensory inte-gration processes in conditions of semantic congruence andbehavioral facilitation to multisensory processes in conditionscontaining cross-modal semantic conflict and reductions inperformance.

MATERIALS AND METHODS

Participants

Eighteen young adults (9 males, ages 19–25 yr, mean � SD: 24.17 �3.9 yr), all right-handed with healthy neurological histories, normal tocorrected-to-normal vision and hearing, and an average of 16.39 yr ofeducation, participated in this study. The study was approved by thejoint Baycrest Centre-University of Toronto Research Ethics Com-mittee, and the rights and privacy of the participants were observed.All participants gave formal informed consent before the experimentand received monetary compensation for participation.

Stimuli

Stimuli were selected to have semantically congruent auditory andvisual representations. Two types of stimuli, animate and inanimate,were used in the study. Items were selected from four distinctcategories: 1) animals, 2) musical instruments, 3) automobiles, and 4)household objects. The first category of stimuli was labeled as“animate,” whereas the remaining three categories were considered“inanimate” objects.

Black-and-white line drawings, selected from the Snodgrass andVanderwart (1980), served as the visual stimuli. All visual stimuliwere matched according to size (in pixels), brightness, and contrast.Participants sat in an upright position and viewed the visual stimuli ona back-projection screen that subtended �30 degrees of visual anglewhen they were seated 70 cm from the screen. Semantically relatednonspeech, complex sounds were matched in terms of loudness bycomputing the mean root mean square value. Each complex soundwas assigned the mean amplitude; thus louder sounds were reduced,whereas softer ones were amplified. Complex sounds were deliveredbinaurally at an intensity level of 60 dB HL based on the audiometricmean across both ears. Binaural auditory stimuli were presented via anOB 822 Clinical Audiometer through ER30 transducers (EtymoticResearch, Elk Grove, IL) and connected with a 1.5-m length ofmatched plastic tubing and foam earplugs to the participants’ ears.

Complex sounds and line drawings were paired to create cross-modal congruent stimulus combinations in which the AV stimulimatched semantically (e.g., picture of a lion paired with the sound ofa roar or picture of an ambulance car paired with a siren). Incongruentcross-modal stimulus combinations, on the other hand, were createdby randomly pairing complex sounds with visual objects from distinctcategories; thus a complex sound or a picture from the animatecategory was always paired with a picture or complex sound from theinanimate category. In summary, four stimulus types were employed:1) auditory unimodal (A), 2) visual unimodal (V), 3) cross-modalcongruent (AV�), and 4) cross-modal incongruent (AV�).

To ensure that the complex sounds were easily nameable andidentifiable, we assessed accuracy values and RTs for each stimulusexemplar in an initial behavioral pilot. Five young adults (mean age:26 yr) participated in this initial pilot. Complex sounds were excludedif detection accuracy levels fell below 75% and RTs exceeded 2 SDabove the mean RT values for each individual subject. Furthermore,after behavioral testing, we also asked participants to rate complex

sounds based on their recognizability. On the basis of the behavioralfindings and the postexperiment questionnaire results, we excludedseveral complex sounds along with their visual counterparts. Thus, foreach animate or inanimate category, 30 different exemplars from eachsensory modality (auditory and visual) were selected because theywere unambiguously categorized. See Table 1 for a complete list ofthe visual and auditory stimuli used in the experiment. In total, 60animate stimuli (30 auditory and 30 visual) and 60 inanimate stimuli(30 auditory and 30 visual) were used.

Procedure

Each stimulus or stimulus pair was presented for 400 ms; for theauditory stimuli, the 400-ms interval also included a 5-ms fall and risetime. The time interval between the end of the stimulus presentationand the beginning of the next trial varied between 2 and 4 s (equiprob-able). See Fig. 1 for an illustration of the paradigm.

Multisensory facilitation, the RT gain following semantically con-gruent AV stimuli, was assessed using a simple detection and a featureclassification task. During the detection task, participants were in-structed to respond (left index finger response) as quickly as possibleto any trial type: unimodal A, unimodal V, cross-modal congruent(AV�), and cross-modal incongruent (AV�). Forty random presen-tations of each stimulus type and a total of 160 trials were used in thiscondition. Therefore, because we used 30 exemplars for each trialtype, certain auditory or visual stimuli were presented more than once.

In the semantic classification task, participants were required tomake animacy or inanimacy judgments following unimodal A, uni-modal V, and cross-modal congruent AV� trial types (left index andmiddle finger responses for animate and inanimate judgments, respec-tively). The correspondence between the response keys and the type ofstimulus presentation was counterbalanced between participants.

Table 1. List of auditory and visual stimuli used in the experiment

Animate Inanimate

Musical instrumentsCow ViolinPig GuitarSheep HarpLamb BanjoDog TrumpetCat BellHorse ClarinetDonkey PianoChicken OrganTurkey DrumRooster Cymbal

Automobiles/transportationCarnival TrainSparrow ShipGoose CarOwl AmbulancePenguin Fire truckWhale MotorcycleDolphin icycleFly AirplaneCricket Helicopter

Household objectsFrog Frying panSnake ForkCrocodile SinkLion BottleBear HairdryerWolf DoorTiger TelephonePuma Tea kettleElephant CameraMonkey Clock

2898 MULTISENSORY FACILITATION AND CROSS-MODAL CONFLICT


by 10.220.32.247 on Septem


Dow

nloaded from


Forty randomized presentations of each stimulus type and a total of120 trials were used in this condition.

To assess cross-modal conflicts, we asked participants to indicatewhether auditory and visual signatures within cross-modal presenta-tions were semantically congruent or not. As such, in the congruencytask, participants discriminated between congruent and incongruentcross-modal presentations by pressing one of two responses (leftindex and middle finger response keys). Forty random presentations ofeach stimulus type were used. The correspondence between theresponse keys and the type of stimulus presentation was counterbal-anced between participants to control for any finger dominance effectson response latencies.

Presentation software (version 10.3; Neurobehavioral Systems, SanFrancisco, CA; http://www.neurobs.com/) was used to control visualand auditory stimulus delivery and to record participants’ responselatency and accuracy.

MEG Recordings

The MEG was recorded in a magnetically shielded room at theRotman Research Institute, Baycrest Centre, using a 151-channelwhole head neuromagnetometer (OMEGA; VSM Medtech, Vancou-ver, Canada). MEG collection was synchronized to the onset of eachvisual stimulus by recording the luminance changes of the screen witha photodiode. With respect to the auditory stimuli, the MEG datacollection was synchronized to the onset of the auditory soundenvelope.

Participants’ head positions within the MEG scanner were deter-mined at the start and at the end of each recording block usingindicator coils placed on nasion and bilateral preauricular points.These fiducial points established a head-based Cartesian coordinate

system for preprocessing and analysis of the MEG data. Threeparticipants were eliminated from the analysis due to excessive headmotion during the MEG scan. Thus MEG data from a total of 15participants were preprocessed and subsequently analyzed.

Neuromagnetic activity was sampled at a rate of 1,250 Hz and wasrecorded continuously in 6 experimental blocks (i.e., 3 experimentalconditions repeated once) of 15 min recording time each. Participantstook a minimum 5-min break between each block. We divided theconditions into 15-min blocks to maximize the total number of trialsfor each condition while also keeping the blocks short to minimize theamount of head motion in the MEG scanner. The simple detectioncondition was always presented first; however, the order of the rest ofthe experimental condition blocks was randomly assigned to eachparticipant.

MEG Preprocessing

Third-gradient noise correction was applied to the continuous MEGdata. Afterwards, the MEG data were parsed into epochs including a200-ms prestimulus and 1,000-ms poststimulus activity window, andDC offsets were removed from the entire epoch. Finally, MEG datawere band-pass filtered between 0.1 and 55 Hz and averaged across alltrial types.

Neuromagnetic activity was also coregistered to each participant’sindividual structural MRI. To constrain the sources of activation toeach participant’s head shape and structural anatomy, we acquiredMRI scans for each participant using a 3.0T Siemens Tim MAGNE-TOM Trio MRI scanner (syngo.MR software; Siemens Medical,Erlangen, Germany) with 12-channel head coil. All participants’structural MRIs and MEG source data were spatially normalized tothe Talairach standard brain using Analysis of Functional Neuroim-aging software (AFNI; Cox 1996).

MEG Data Analysis

Event-related SAM analysis. The “nonlinear beamformer” or syn-thetic aperture magnetometry (SAM) technique was used to model thesource of the measured magnetic field (Robinson and Vrba 1998;Sekihara et al. 2001). SAM minimizes power or the variance of themeasured MEG signals such that signals emitted from sources outsideeach specified voxel are suppressed (Brookes et al. 2007; Cheyne etal. 2007). This enables one to display simultaneously active sources atmultiple sites, provided that they are not perfectly synchronized. Toobtain spatial precision without integrating power over long temporalwindows, we used an event-related version of the SAM analysistechnique, event-related SAM or ER-SAM, introduced by Cheyne etal. (2006) to identify evoked brain responses from unaveraged single-trial data.

Despite their advantages in terms of spatial resolution and local-ization accuracy, the performance of beamformers can be reduced inthe presence of sources with high temporal correlations. However,beamformers used to localize time-locked brain activity, includingER-SAM, have been shown to be fairly robust in the presence ofsources with small to moderate correlations, exhibiting only smalldecreases in amplitude in the presence of such sources (Gross et al.2001; Hadjipapas et al. 2005; Sekihara et al. 2002). It is important toconsider, however, that the proximity between highly correlatedsources may reduce estimated power in such sources. Quraan andCheyne (2010) used simulations of fully correlated bilateral sources inthe auditory cortex as well as correlated bilateral sources in striate andextrastriate areas with random Gaussian noise and demonstrated thatsource cancellations were less likely to occur at signal-to-noise levelstypical of neural sources in current MEG systems. Another reason forselecting ER-SAM over more traditional beamformers is that it canlocalize sources that are time-locked to stimulus events even in thepresence of artifacts and instrumental noise (Cheyne et al. 2007).

Fig. 1. Experimental design. Four possible stimulus combinations were used inthis study: 1) unimodal visual (V; example, picture of a violin), 2) unimodalauditory (A; example, sound of a “roar”), 3) cross-modal congruent orsimultaneous auditory and visual stimuli that were matched semantically(AV�; example, picture of a bird matched with a corresponding “chirp”sound), and 4) cross-modal incongruent or simultaneous auditory and visualstimuli that were semantically mismatched (AV�; example, picture of a lionmatched with a “siren” sound). Each stimulus or stimulus pair was presentedfor 400 ms; for the auditory stimulus, the 400-ms interval also included a 5-msfall and rise time. The time interval between the end of the stimulus presen-tation and the beginning of the next trial (ITI) varied between 2 and 4 s(equiprobable).



by 10.220.32.247 on Septem


Dow

nloaded from


Similarly to previous beamforming approaches, the ER-SAM anal-ysis uses the individual trials of each condition and the forwardsolution for modeling optimal current direction to calculate a spatialfilter for each voxel using the minimum-variance beamforming algo-rithm (Cheyne et al. 2006). The spatial filter we used included 72brain regions of interest adapted from the regional map coarse par-cellation scheme of the cerebral cortex proposed by Kotter andcolleagues (Bezgin et al. 2008; Kotter and Wanke 2005). See Table 2for a complete listing of the brain regions used with their respectiveTalairach coordinates. Each brain region was defined by a three-dimensional position vector and consisted of a unique set of sensorcoefficients that constituted a weighting matrix. Thus all brain regionswere constrained to be the same size.

The MEG data were then projected through this spatial filter to givea measure of current density, as a function of time, in the target brainregion. Because this source time series was calculated using aweighted sum of the MEG sensors, it had the same millisecond timeresolution as the original MEG sensor data. To enhance the spatialprecision of this technique, we used the participants’ structural MRIsto constrain the ER-SAM images to each participant’s individual MRIand to allow for spatial normalization and group averaging in stereo-taxic space. The individual functional maps were overlaid on theindividual participant’s MRI based on coregistration with the indica-tor coils placed on the nasion and bilateral preauricular points. Thefunctional data were then transformed to the standard Talairach-Tournoux space using the same transformation applied to the struc-tural MRI (AFNI software; Cox 1996).

PLS analysis. We used partial least squares (PLS) analysis (McIn-tosh et al. 1996) to examine neuromagnetic brain activity across all 72brain regions of interest as a function of task demands, namely,multisensory facilitation and cross-modal conflict. The term “partialleast squares” refers to the computation of an optimal squares fit topart of a covariance structure that is attributable to the experimentalmanipulations. PLS applied to MEG data is conceptually analogous tothe analysis of MEG difference waveforms, because it identifiestask-related differences in amplitude across all MEG sources byderiving the optimal least squares contrasts that code for the taskdifferences. Because PLS performs this computation across the entiredataset in time and space simultaneously, there is no need to specifya priori MEG sources or time intervals for the analysis.

In the original version of PLS, singular value decomposition (SVD)was used to identify the strongest effects in the data. For severalanalyses discussed in this study, we used a nonrotated version of taskPLS, in which a priori contrasts restricted the spatiotemporal patternsderived from PLS. This version of PLS has the advantage of allowingdirect assessment of hypothesized experimental effects. Task differ-ences were examined across the entire epoch, including the prestimu-lus baseline (�0.2 s) and the poststimulus interval (1 s).

The relationship between two sets of matrices is analyzed: the firstpertains to the experimental design (or to the performance measures),whereas the second contains MEG source activity across all timepoints. The relationship between the two blocks of data is stored in across-product matrix. Application of SVD decomposes this matrixinto three new matrices: 1) task saliences, 2) singular values, and3) source saliences. The task saliences indicate the experimentaldesign profiles that best characterize the given matrix. The sourcesaliences reflect the MEG sources that characterize the correspondingtask pattern across space (expressed across a collection of MEGsources) and time (expressed across all time points included in theanalysis). Task and source saliences are linked together via singularvalues, which represent the square root of the eigenvalues. Thecross-product between the task saliences and original matrix creates“brain scores,” which reflect the expression of the task contrast acrossall participants (see Figs. 4—7).

Statistical assessment. For statistical assessment of task effects,two complementary resampling techniques were employed. First,permutation tests assessed whether the task saliences were signifi-

cantly different from random noise. Second, the reliability of eachsource contribution was assessed using a bootstrap estimation ofstandard errors for the MEG source saliences. The primary purpose ofthe bootstrap is to determine those portions of the source waveformsthat show reliable experimental effects across subjects. Importantly,with the use of bootstrap estimation of standard errors, no correctionfor multiple comparisons was necessary, because the source salienceswere calculated in a single mathematical step on the whole brain atonce (McIntosh et al. 1996; McIntosh and Lobaugh 2004). These tworesampling techniques provide complementary information about thestatistical strength of the task effects and its reliability across partic-ipants. Statistical evaluation of task effects was performed using anoptimal number of 500 permutations (cf. Nichols and Holmes 2002)and 300 bootstrap iterations (cf. Efron and Tibshirani 1986; McIntoshet al. 1996).

RESULTS

Behavioral Results

Performance accuracy. In the simple detection condition,accuracy levels were at ceiling in all four trial types (i.e., AV�,AV�, A, and V). In the animacy condition, a larger number oferrors were observed for unimodal A compared with AV� andunimodal V trial types [F(2,13) � 207.33, P � 0.001]. Fur-thermore, in the congruency condition, participants showedmore false alarms in response to AV� trial types. In otherwords, participants were more likely to classify AV stimuli asincongruent in cases of semantic congruence [F(1,14) �131.81, P � 0.0001]. See Table 3 for means and SD values inthe three conditions and four trial types.

Response times. MULTISENSORY FACILITATION. Only correctresponses were used in the RT data analysis. In multisensoryfacilitation conditions (simple detection and semantic classifi-cation), RTs were analyzed using a 2 � 3 repeated-measuresanalysis of variance (ANOVA) in which trial type (i.e., A, V,and AV�) and condition (i.e., simple detection and semanticclassification) were the within-subject factors. First, a maineffect of condition was observed [F(1, 14) � 478.96, P �0.001] with RTs for semantic classification conditions beinglonger (mean � SD: 657 � 17 ms) than RTs in the simpledetection condition (289 � 14 ms). Second, a main effect oftrial type was identified [F(2, 28) � 135.83, P � 0.001] withunimodal A trial types showing significantly longer RTs thanunimodal V and cross-modal trial types across all three con-ditions (see Table 3 for means � SD values in each conditionand trial type; Fig. 2, A and B).

Finally, an interaction between conditions and trial typeswas also found [F(2, 28) � 29.90, P � 0.0001] reflectingsignificant behavioral facilitation to cross-modal comparedwith unimodal A presentations in both semantic classificationand simple detection conditions (see Table 4 for an overview ofthe repeated-measures ANOVA results). In the simple detec-tion condition, post hoc t-tests suggested that RTs to unimodalV trial types were also significantly slower than RTs to AV�trial types (t � 11.47, P � 0.001) and RTs to AV� trial types(t � 11.35, P � 0.001). In conclusion, multisensory facilitationwas captured across both auditory and visual modalities in thesimple detection condition and in the auditory modality only inthe semantic classification condition.

CROSS-MODAL CONFLICTS. We examined the effects of cross-modal incongruence on performance by comparing congruent(AV�) and incongruent (AV-) trial types in both simple



by 10.220.32.247 on Septem


Dow

nloaded from


Table 2. Regional map coordinates with reference to Talairach-Tournoux

Cluster X Y Z Region BA Hemisphere

1 0 32 24 Anterior cingulate cortex 32 Midline2 0 �32 24 Posterior cingulate cortex 23 Midline3 0 �48 12 Retrosplenial cingulate cortex 30 Midline4 0 16 �8 Aubgenual cingulate cortex 25 Midline5 �40 �14 4 A1 (primary auditory) Left6 �60 �14 4 A2 (secondary auditory) 22 Left7 �36 8 56 Frontal eye fields 6 Left8 �36 16 �4 Anterior insula 13 Left9 �36 �8 �4 Claustrum Left

10 �24 �24 56 M1 (primary motor) 4 Left11 �44 �48 20 Inferior parietal cortex 40 Left12 �44 �64 28 Angular gyrus 39 Left13 �8 �64 54 Precuneus 7 Left14 �28 �56 54 Superior parietal cortex 7 Left15 �48 32 12 Centrolateral prefrontal cortex 46 Left16 �48 36 32 Dorsolateral prefrontal cortex 9 Left17 �8 36 40 Dorsomedial prefrontal cortex 8 Left18 �8 48 20 Medial prefrontal cortex 10 Left19 �24 44 �20 Orbitofrontal cortex 11 Left20 �24 64 4 Frontal polar 10 Left21 �48 32 �8 Ventrolateral prefrontal cortex Left22 �28 �16 �16 Parahippocampal cortex Left23 �28 0 60 Dorsolateral premotor cortex 6 Left24 �4 0 60 Medial premotor cortex 6 Left25 �44 4 24 Ventrolateral premotor cortex 9 Left26 �16 �28 4 Pulvinar Left27 �40 �28 64 S1 (primary somatosensory) 3 Left28 �56 �16 16 S2 (secondary somatosensory) 43 Left29 �64 �24 �12 Middle temporal cortex 21 Left30 �64 �24 �24 Inferior temporal cortex 20 Left31 �52 12 �28 Temporal pole 38 Left32 �52 �4 �8 Superior temporal cortex 22 Left33 �32 �28 �28 Ventral temporal cortex Left34 �8 �8 4 Thalamus (ventral lateral nucleus) Left35 �4 �84 �4 V1 (primary visual) Left36 �4 �96 8 V2 (secondary visual) Left37 �20 �88 20 Cuneus 18 Left38 �20 �84 �12 Fusiform gyrus 19 Left39 40 �14 4 A1 (primary auditory) Right40 60 �14 4 A2 (secondary auditory) 22 Right41 36 8 56 Frontal eye fields 6 Right42 36 16 �4 Anterior insula 13 Right43 36 �8 �4 Claustrum Right44 24 �24 56 M1 (primary motor) 4 Right45 44 �48 20 Inferior parietal cortex 40 Right46 44 �64 28 Angular gyrus 39 Right47 8 �64 54 Precuneus 7 Right48 28 �56 54 Superior parietal cortex 7 Right49 48 32 12 Centrolateral prefrontal cortex 46 Right50 48 36 32 Dorsolateral prefrontal cortex 9 Right51 8 36 40 Dorsomedial prefrontal cortex 8 Right52 8 48 20 Medial prefrontal cortex 10 Right53 24 44 �20 Orbitofrontal cortex 11 Right54 24 64 4 Frontal polar 10 Right55 48 32 �8 Ventrolateral prefrontal cortex Right56 28 �16 �16 Parahippocampal cortex Right57 28 0 60 Dorsolateral premotor cortex 6 Right58 4 0 60 Medial premotor cortex 6 Right59 44 4 24 Ventrolateral premotor cortex 9 Right60 16 �28 4 Pulvinar Right61 40 �28 64 S1 (primary somatosensory) 3 Right62 56 �16 16 S2 (secondary somatosensory) 43 Right63 64 �24 �12 Middle temporal cortex 21 Right64 64 �24 �24 Inferior temporal cortex 20 Right65 52 12 �28 Temporal pole 38 Right66 52 �4 �8 Superior temporal cortex 22 Right67 32 �28 �28 Ventral temporal cortex Right68 8 �8 4 Thalamus (ventral lateral nucleus) Right69 4 �84 �4 V1 (primary visual) Right

Continued



by 10.220.32.247 on Septem


Dow

nloaded from


detection and congruency conditions. In the detection condi-tion, we expected to find no differences between congruent andincongruent presentations. However, in the congruency condi-tion, we anticipated that RTs would be slower as participantswere required to assess the degree of mismatch between theauditory and visual stimuli embedded within cross-modalpresentations.

We found a significant interaction between trial types andconditions [F(1,14) � 37.39, P � 0.001]. As predicted, cross-modal congruent and incongruent trial types did not differ inthe simple detection condition; however, in the congruencycondition, incongruent compared with congruent trial typesexhibited significantly longer RTs (Fig. 2C). In the congruencycondition, post hoc t-tests revealed significant differences be-tween congruent and incongruent AV trial types [t(14) � 6.19,P � 0.0001]. Participants were slower to respond to incongru-ent AV trial types by �140 ms. Although they were slower,participants were also more accurate in categorizing incongru-ent compared with congruent presentations. Because we wereprimarily interested in task differences in RTs, we instructedparticipants to respond as fast as possible to each stimulus type.Thus, in the congruency condition, responses to congruent AVstimuli were faster, but also less accurate, than incongruentones. Performance accuracy was 4% lower in congruent com-pared with incongruent trials (t � 11.4, P � 0.0001) averagingat 85% and 89%, respectively.

Cumulative distribution functions. To investigate the behav-ioral effects of multisensory integration further, we contrastedcross-modal trial types (i.e., AV� and AV-) against the in-equality model proposed by Miller (1982). The group-averagedprobability for each cross-modal trial type was analyzedagainst the predictions of the model based on the unimodal

Table 2. —Continued

Cluster X Y Z Region BA Hemisphere

70 4 �96 8 V2 (secondary visual) Right71 20 �88 20 Cuneus 18 Right72 20 �84 �12 Fusiform gyrus 19 Right

Coordinates are included for the 72 brain regions of interest adapted from the regional map coarse parcellation scheme of the cerebral cortex proposed by Kotterand Wanke (2005). Brodmann areas (BA) were determined using the brain atlas of Talairach and Tournoux (1988), because these landmarks provide moremeaningful representations of the magnetoencephalography source locations.

Table 3. Descriptive statistics for the two dependent variables:accuracy and RT

Variable Mean SD

AccuracySimple detection AV� 98.33 0.90Simple detection AV� 97.33 1.35Simple detection A 98.73 0.46Simple detection V 97.07 1.28Semantic classification AV� 97.20 0.94Semantic classification A 90.40 1.68Semantic classification V 97.53 1.06Congruency AV� 84.87 0.48Congruency AV� 89.93 0.30

RTSimple detection AV� 254.54 47.77Simple detection AV� 249.72 47.41Simple detection A 330.18 80.37Simple detection V 284.59 46.25Semantic classification AV� 592.91 70.73Semantic classification A 779.42 75.87Semantic classification V 598.79 83.25Congruency AV� 957.835 172.0379Congruency AV� 1,099.843 192.0922

Values are percentages for accuracy and milliseconds for response time(RT) for unimodal auditory (A), unimodal visual (V), cross-modal congruent(AV�), and cross-modal incongruent (AV�) trial type statistics.

Fig. 2. Response times (RTs; in ms) in simple detection (A), semantic featureclassification (B), and congruency conditions (C). Errors bars represent SE.



by 10.220.32.247 on Septem


Dow

nloaded from


distributions. We generated subject-specific cumulative distri-bution functions (CDFs) for each trial type by dividing the timeseries into smaller, 5-ms time bins, similar to the procedureused by Laurienti and colleagues under similar task demands(Laurienti et al. 2006; Peiffer et al. 2007). The race model CDFwas generated for each subject and averaged at each time bin.Paired t-tests were used to compare multisensory responsesagainst the statistical facilitation predictions of the race modelbased on the unimodal RT distributions for each participant. Inthe simple detection condition, the probability of detectingfaster RTs following cross-modal presentations exceeded the

probability predicted by statistical facilitation associated withredundant unisensory stimuli (t � 69.92, P � 0.001). Simi-larly, in the semantic classification condition, multisensoryresponses were also faster than those predicted by the racemodel (t � 19.25, P � 0.003). Note that the probabilities ofdetecting shorter RTs in AV� trials were not larger than theprobabilities of detecting short RTs in unimodal visual trials inthe semantic classification condition, further indicating thatsemantic classification of visual objects does not benefit fromconcurrent auditory presentations (see Fig. 3B).

MEG Results

Multisensory processing. In both simple detection and se-mantic classification conditions, significant differences be-tween cross-modal and unimodal auditory presentations weredetected [first-order latent variable (LV1) � 48.94, P � 0.001;Fig. 4A]. Multisensory integration, or increased MEG sourceactivity in response to cross-modal compared with unimodalauditory stimuli, was captured in the left temporal pole (BA38) and the left angular gyrus (BA 39) between 100–300 and200–400 ms, respectively (Fig. 4B). Cross-modal and uni-modal V trial types also differed significantly (LV1 � 52.43,P � 0.001; Fig. 4C). Multisensory processes were captured inthe left cuneus (BA 18) and the left IPL (BA 40) between100–300 and 200–400 ms, respectively, with these sourcesshowing larger amplitude modulations to cross-modal stimulicompared with visual ones. (Fig. 4D).

Our initial prediction that posterior parietal cortices re-sponded preferentially to cross-modal trial types was alsosupported by an analysis that compared cross-modal (AV�and AV�) with unisensory presentations (LV1 � 44.96, P �0.05; Fig. 5A). When comparing cross-modal stimuli with theirrespective unisensory components, we found evidence of mul-tisensory integration in the left precuneus (BA 7) and the rightprimary motor cortex (BA 4) between 100–200 and 300–600ms after stimulus onset, respectively (Fig. 5B).

EFFECTS OF SEMANTIC EVALUATION. The multisensory integra-tion neural processes described above were captured in both

Table 4. Repeated-measures ANOVA results and descriptive statistics

Effect F Hypothesis df Error df P Partial �2

Condition 478.9638 1.0000 14.0000 0.0000 0.9716Trial type 135.8296 2.0000 13.0000 0.0000 0.9408Condition �

trial type 29.89567 2.0000 13.0000 0.0002 0.7345

Effect Mean SD

95% ConfidenceInterval

Lowerbound

Upperbound

ConditionSimple detection 289.77 14.78 258.07 321.47Semantic classification 657.04 17.58 619.34 694.74

Trial typeAV� 423.72 12.92 396.00 451.44A 554.80 15.58 521.39 588.21V 441.69 15.64 408.15 475.24

Condition � trial typeSimple detection AV� 254.54 12.33 228.08 280.99Simple detection A 330.18 20.75 285.68 374.69Simple detection V 284.59 11.94 258.98 310.21Semantic classification AV� 592.91 18.26 553.74 632.08Semantic classification A 779.42 19.59 737.41 821.43Semantic classification V 598.79 21.49 552.69 644.89

Values are results of repeated-measures analysis of variance (ANOVA) andother descriptive statistics as indicated. df, Degrees of freedom.

Fig. 3. Cumulative probability density function (CDF) for cross-modal trial types and multisensory responses predicted by the race (or inequality) model.Multisensory and calculated (predicted) race model CDFs are shown. Multisensory responses were faster than those predicted by the race model in both detection(A) and semantic classification conditions (B). Note that in the semantic classification condition, multisensory facilitation was only detected when auditory stimuliwere accompanied by visual ones, rather than vice versa.



by 10.220.32.247 on Septem


Dow

nloaded from


simple detection and semantic classification conditions. In thesemantic classification condition, increased amplitude modu-lations in response to semantically congruent cross-modalstimuli were observed in a distinct set of brain regions. Inaddition to multisensory responses in sensory-specific andposterior parietal brain regions, increased activity in posteriorcingulate and superior temporal cortices were observed asparticipants were required to classify cross-modal trial typesinto animate and inanimate categories (LV1 � 48.13, P �0.05; Fig. 6A). The posterior cingulate cortex and the left STSshowed larger amplitudes in response to AV� trial typesbetween 100–300 and 500–700 ms, respectively (Fig. 6B).

The effects of semantic classification across cingulate andsuperior temporal regions were specific to semantically con-gruent cross-modal presentations. The specificity of cingulate

and temporal activity in response to cross-modal stimuli wastested explicitly by assessing the impact of semantic categori-zation on unisensory processing. In visual-only trial types,there was a trend toward increased activity in the left fusiformgyrus as participants classified pictures into animate or inani-mate categories. A similar trend was observed in the auditorymodality in the insula. These differences, however, were notsignificant by permutation tests. Thus we concluded that pos-terior cingulate and left STS activity between 100–300 and500–700 ms was specific to cross-modal presentations in thesemantic classification condition.

EFFECTS OF CROSS-MODAL CONFLICT. Compared with the mul-tisensory facilitation effects in the simple detection conditions,performance decreases were observed in AV� and AV� trialsas participants were required to determine the degree of con-

Fig. 4. Cross-modal compared with unimodal auditory (A and B) and unimodal visual processing (C and D) in multisensory facilitation conditions. The sourcewaveforms were derived from the Talairach coordinates displayed at left. The bootstrap ratios above the source waveforms reflect the positive expression of thetask contrast (A and C), i.e., larger amplitude modulations for cross-modal compared with unimodal auditory and unimodal visual trial types. The left temporalpole and angular gyrus (AG) showed increased activity in response to cross-modal compared with unimodal auditory trial types between 100–300 and 200–600ms, respectively (B). The left cuneus and inferior parietal lobule (IPL) showed increased activity in response to cross-modal compared with unimodal visual trialtypes between 100–300 and 200–400 ms (D).

22

Fig. 5. Cross-modal compared with unimodal trial types in multisensory facilitation conditions. The source waveforms were derived from the Talairachcoordinates displayed at left. The bootstrap ratios above the source waveforms reflect the positive expression of the task contrast (A), i.e., larger amplitudemodulations for cross-modal compared with unimodal trial types. We found increased activity in the left precuneus and the right primary motor cortex (M1)between 100–200 and 300–600 ms (B).



by 10.220.32.247 on Septem


Dow

nloaded from


gruency between auditory and visual targets. We observedsignificant differences between cross-modal trial types in thesimple detection and the congruency conditions (LV1 � 55.25,P � 0.001; Fig. 7A). Large-amplitude modulations extending

between 200–400 and 500–1,000 ms after stimulus onset werecaptured in the congruency condition when participants wererequired to determine whether cross-modal presentationsmatched semantically.

Cortex

Fig. 6. Effects of semantic feature categorization on multisensory processing. The source waveforms were derived from the Talairach coordinates displayed atleft. The bootstrap ratios below the source waveforms reflect the negative expression of the task contrast (A), i.e., larger amplitude modulations for cross-modaltrial types in the semantic classification condition compared with cross-modal trial types in the simple detection condition. The medial posterior cingulate andleft superior temporal sulcus showed increased activity in response to cross-modal trial types in the semantic classification condition between 100–300 and500–700 ms (B).

Fig. 7. Effects of cross-modal conflict on multisensory processing. The source waveforms were derived from the Talairach coordinates displayed at left. Thebootstrap ratios below the source waveforms reflect the negative expression of the task contrast (A), i.e., larger amplitude modulations for cross-modal trial typesin the congruent condition compared with the simple detection condition, and larger amplitudes for incongruent compared with congruent trial types. The leftparahippocampal cortex (PHC), dorsomedial prefrontal cortex (PFC), premotor cortex (PMC), and bilateral orbitofrontal cortex (OFC) showed increased activityin response to cross-modal trial types in the congruency condition between 200–500 and 600–1,000 ms (B).



by 10.220.32.247 on Septem


Dow

nloaded from


The left parahippocampal cortex (PHC) and the left dorso-medial prefrontal cortex (PFC) showed large amplitude mod-ulations between 200–400 and 400–600 ms in response toAV� and AV� trial types in the congruency condition com-pared with the simple detection condition (Fig. 7B). Further-more, the bilateral orbitofrontal cortex (BA 11) and the leftdorsolateral premotor cortex (PMC) were also associated withcross-modal presentations in the congruency condition be-tween 400–600 and 600–1,000 ms (Fig. 7B). Source activityin the left dorsolateral PMC was positively correlated with RTsacross all trial types in the congruency condition between 500and 1,000 ms after stimulus onset (r � 0.942, 0.945, 0.71, and0.82, P � 0.001, in AV�, AV�, A, and V trial types,respectively).

Within the congruency condition, we also observed differ-ences between congruent and incongruent trial types (LV1 �41.98, P � 0.05; Fig. 8A). The left dorsomedial PFC and theleft centrolateral PFC showed larger responses to incongruentcompared with congruent trial types between 100 and 300 ms(Fig. 8B).

DISCUSSION

Behavior Results

Multisensory facilitation at the behavioral level was ob-served in simple detection and semantic classification condi-tions with faster RTs and improved accuracy for cross-modalcompared with unimodal trial types. Multisensory responses inboth simple detection and semantic classification conditionsalso exceeded the responses predicted by the race model (cf.Fig. 3). In the semantic classification condition, however, thepresentation of the visual target facilitated auditory objectrecognition, but not vice versa. Although motor performancewas slower for unimodal A compared with cross-modal trialtypes, no significant differences between the unimodal visualand cross-modal trial types were observed. Indeed, previousstudies demonstrated an asymmetry between auditory andvisual stimuli with increased dominance of vision over audi-tory perception during object recognition tasks. As demon-strated previously, concurrent visual presentations helped dis-ambiguate auditory object processes more than vice versa(Jaekl and Harris 2009; Yuval-Greenberg and Deouell, 2009).

Although the behavioral results in the semantic classificationcondition indicate dominance of vision over audition, recentERP studies suggest that, similarly to the visual sensory mo-

dality, auditory animacy judgments begin within 100 ms afterstimulus onset (Murray et al. 2006). Similarly, Thorpe et al.(1996) showed that animacy judgments of visual objects in-crease neural activity in sensory-specific areas 100 ms afterstimulus presentation.

In the congruency condition, which assessed the effects ofcross-modal semantic relatedness on classification perfor-mance, slower RTs and an increased number of false alarmswere observed in response to congruent presentations relativeto incongruent ones. When they were faster, participants werealso more likely to judge cross-modal stimuli as congruent.Similar to previous behavioral findings (cf., Laurienti et al.2003, 2004; Noppeney et al. 2008), even after incorrect trialswere excluded, RTs to congruent presentations continued to besignificantly faster than RTs to incongruent presentations.

Distinct Effects of Multisensory Facilitation and Cross-ModalConflict on Neuromagnetic Activity

MEG recordings and ER-SAM analysis allowed us to ex-amine multisensory processes both temporally and spatially.We observed evidence of multisensory integration across sen-sory-specific and posterior parietal sources in both auditory andvisual modalities. These spatiotemporal patterns were capturedin both simple detection and semantic classification conditions.Within 100–200 ms after stimulus onset, cross-modal stimuli,compared with unimodal stimuli, elicited increased amplitudemodulations in left posterior parietal cortices, including the leftprecuneus, left angular gyrus, and the left IPL. MEG sourceactivity in the left temporal pole and angular gyrus increased inresponse to cross-modal presentations relative to unimodalauditory presentations. Although we observed significant sen-sory modality differences in performance, the timing of mul-tisensory responses did not differ between auditory and visualmodalities. Similar to the auditory modality, larger amplitudesto cross-modal compared with unimodal visual trial typesextended between 100–300 and 200–400 ms. In addition,examination of multisensory responses against the responsesevoked by both modality-specific components revealed en-hanced activity in the left posterior parietal sources, includingthe left precuneus, and the right motor cortex between 100 and400 and at 100 ms, respectively, in response to cross-modalstimuli.

Posterior parietal activity has been previously linked tomultisensory processing. Neuroimaging studies using fMRIrecordings to examine multisensory integration processes

Fig. 8. Congruent compared with incongruent cross-modal matching. The source waveforms were derived from the Talairach coordinates displayed at left. Thebootstrap ratios below the source waveforms reflect the negative expression of the task contrast (A), i.e., larger amplitude modulations for cross-modal trial typesin the congruent condition compared with the simple detection condition, and larger amplitudes for incongruent compared with congruent trial types. The leftdorsomedial PFC and the left centrolateral PFC showed increased activity to incongruent compared with congruent trial types between 100 and 300 ms (B).



by 10.220.32.247 on Septem


Dow

nloaded from


showed that posterior parietal regions including IPS, the IPL(BA 40), and the SPL (BA 39) responded preferentially tocross-modal compared with unimodal presentations (Baumannand Greenlee 2007; Bishop and Miller 2008; Calvert et al.2001; Grefkes et al. 2002; Macaluso et al. 2004; Moran et al.2008). Previous ERP studies also found early-amplitude de-flections across sensory-specific channels and late-amplitudedeflections in parietal scalp regions during the presentation ofsemantically and spatially congruent AV stimuli (Giard andPeronnet 1999; Teder-Salejarvi et al. 2002, 2005).

Although multisensory processes were captured bilaterallyin posterior parietal cortices, the difference between cross-modal and unimodal trial types was larger and more stableacross participants in the left hemisphere. This left hemisphericbias in posterior parietal activity during multisensory process-ing is not uncommon. There is growing evidence that whencross-modal activation patterns are not found bilaterally, theyare usually observed in the left hemisphere (cf. Baumann andGreenlee 2007; Calvert et al. 2001; Grefkes et al. 2002;Molholm et al. 2006).

In addition to increased amplitude modulations associatedwith cross-modal stimuli, we also demonstrated that activity incingulate and superior temporal regions increased 100 and 500ms after stimulus onset as participants were required to deter-mine which cross-modal pairs pertained to animate or inani-mate categories. These MEG source activation patterns werespecific to cross-modal trial types, because they were notdetected when contrasting unisensory responses across simpledetection and semantic classification conditions.

Previous neuroimaging studies also showed that simultane-ously presented complex sounds and semantically-related vi-sual stimuli recruited anterior cingulate and adjacent medialprefrontal cortices (Laurienti et al. 2003). Furthermore, con-sistent with our current results, Beauchamp et al. (2004) andvan Atteveldt et al. (2010) showed that blood oxygen level-dependent activity in the posterior STS and the superior tem-poral cortex was specifically associated with the processing offamiliar, semantically related cross-modal stimuli. The presentstudy extends this line of research by demonstrating that neuralactivity related to semantic feature classification begins 100 msafter stimulus onset in cingulate regions and extends to supe-rior temporal regions 400 ms later in response to cross-modalstimulus presentations only.

In cross-modal matching trial types, activity in the left PHCincreased between 200–400 and 600–1,000 ms across bothcongruent and incongruent cross-modal pairs. Furthermore, thebilateral orbitofrontal cortex (BA 11) and the left PMC showedsimilar task-related activation patterns at later processingstages, between 500 and 1,000 ms after stimulus onset. We alsofound that increased activity in the left PMC predicted slowerperformance in response to cross-modal stimuli as participantsassessed the degree of congruence between their auditory andvisual counterparts. Because response latencies in the congru-ency condition averaged at �858 ms, increased premotoractivity in this condition may reflect motor response prepara-tion. On the other hand, recent research has shown that leftPMCs respond preferentially to inanimate objects such as tools(Maravita et al. 2002; Weisberg et al. 2007). Although we werenot interested in differences between animate and inanimatestimuli, we compared the two stimulus categories and found a

lack of significant differences between animate and inanimatetrial types in motor and premotor sources.

Within cross-modal trial types, the left dorsomedial andcentrolateral prefrontal cortices responded preferentially toincongruent cross-modal combinations compared with congru-ent ones 100–400 ms after stimulus onset. Previous fMRIstudies that examined the effects of semantic congruence onmultisensory integration suggested that the inferior frontalcortex activated in response to cross-modal stimuli when au-ditory and visual representations were semantically mis-matched. Noppeney et al. (2008, 2010) used pictures of ani-mate (animals) and inanimate (man-made) objects concurrentlypresented with matching and nonmatching complex sounds orspoken words. Stronger effects of semantic relatedness wereobserved in the posterior STS and middle temporal gyrus inresponse to words. However, only medial and inferior frontalcortices showed incongruency effects for both spoken wordsand complex sounds when paired with animate or inanimatepictures. In line with these results, Belardinelli et al. (2004)also demonstrated that the inferior frontal cortex respondedmore strongly to incongruent cross-modal combinations ofcomplex stimuli (Belardinelli et al. 2004).

The present study extends previous examinations of cross-modal conflict by demonstrating that detecting the degree ofsemantic AV incongruence elicits increased activity in medialand centrolateral prefrontal cortices as early as 100 ms afterstimulus onset. Furthermore, as participants are instructed toidentify the degree of cross-modal semantic relatedness, addi-tional sources, namely, left PHC, show large-amplitude mod-ulations as early as 200 ms after stimulus onset. After �250ms, premotor and bilateral orbitofrontal cortices show similartask-related amplitude modulations with premotor activity re-flecting motor response preparation.

Conclusion

Multisensory integration, or the distinct neural processesassociated with cross-modal relative to unimodal events,was captured in sensory-specific and posterior parietal brainregions within 100 and 200 ms after stimulus onset. Cingu-late and superior temporal cortices responded preferentiallyto cross-modal presentations between 100 and 500 ms asparticipants classified cross-modal pairs into animate oranimate categories. Finally, in trials that required partici-pants to detect the level of cross-modal incongruence, me-dial and centrolateral prefrontal cortices showed early am-plitude modulations in response to incongruent pairs com-pared with congruent ones. These results suggest thatintegration of semantically related complex sounds and pic-tures unfolds in multiple stages and across distinct parietaland medial frontal cortical regions: first, sensory-specificand posterior parietal neurons respond preferentially tocross-modal stimuli irrespective of task demands and asearly as 100 ms after stimulus onset; second, regions insuperior temporal and posterior cingulate cortices respondto cross-modal stimuli during semantic feature classificationtrials between 100 and 500 ms; and finally, within 100 – 400ms and 400 – 800 ms after stimulus onset, parahippocampal,medial prefrontal, and orbitofrontal regions process cross-modal conflicts and respond to cross-modal stimuli whencomplex sounds and pictures are semantically mismatched.



by 10.220.32.247 on Septem


Dow

nloaded from


GRANTS

This research was supported by grants from the Canadian Institutes ofHealth Research and the J. S. McDonnell Foundation to A. R. McIntosh. A. O.Diaconescu is supported by the Natural Sciences and Engineering ResearchCouncil of Canada.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

A.O.D., C.A., and A.R.M. conception and design of research; A.O.D.performed experiments; A.O.D. analyzed data; A.O.D. and A.R.M interpretedresults of experiments; A.O.D. prepared figures; A.O.D. drafted manuscript;A.O.D., C.A., and A.R.M. edited and revised manuscript; A.O.D., C.A., andA.R.M. approved final version of manuscript.

REFERENCES

Baumann O, Greenlee MW. Neural correlates of coherent audiovisualmotion perception. Cereb Cortex 17: 1433–1443, 2007.

Beauchamp MS, Nath AR, Pasalar S. fMRI-Guided transcranial magneticstimulation reveals that the superior temporal sulcus is a cortical locus of theMcGurk effect. J Neurosci 30: 2414–2417, 2010.

Belardinelli MO, Sestieri C, Di Matteo R, Delogu F, Del Gratta C, FerrettiA, Caulo M, Tartaro A, Romani GL. Audio-visual crossmodal interac-tions in environmental perception: An fMRI investigation. Cogn Process 5:167–174, 2004.

Bezgin G, Wanke E, Krumnack A, Kotter R. Deducing logical relationshipsbetween spatially registered cortical parcellations under conditions of un-certainty. Neural Netw 21: 1132–1145, 2008.

Bishop CW, Miller LM. A multisensory cortical network for understandingspeech in noise. J Cogn Neurosci 21: 1790–1805, 2009.

Brookes MJ, Stevenson CM, Barnes GR, Hillebrand A, Simpson MI,Francis ST, Morris PG. Beamformer reconstruction of correlated sourcesusing a modified source model. Neuroimage 34: 1454–1465, 2007.

Calvert GA, Hansen PC, Iversen SD, Brammer MJ. Detection of audio-visual integration sites in humans by application of electrophysiologicalcriteria to the BOLD effect. Neuroimage 14: 427–438, 2001.

Chen YC, Spence C. When hearing the bark helps to identify the dog:semantically-congruent sounds modulate the identification of masked pic-tures. Cognition 114: 389–404, 2010.

Cheyne D, Bakhtazad L, Gaetz W. Spatiotemporal mapping of corticalactivity accompanying voluntary movements using an event-related beam-forming approach. Hum Brain Mapp 27: 213–229, 2006.

Cheyne D, Bostan AC, Gaetz W, Pang EW. Event-related beamforming: arobust method for presurgical functional mapping using MEG. Clin Neuro-physiol 118: 1691–1704, 2007.

Cox RW. AFNI: software for analysis and visualization of functional magneticresonance neuroimages. Comput Biomed Res 29: 162–173, 1996.

Diederich A, Colonius H. Bimodal and trimodal multisensory enhancement:effects of stimulus onset and intensity on reaction time. Percept Psychophys66: 1388–1404, 2004.

Doehrmann O, Naumer MJ. Semantics and the multisensory brain: howmeaning modulates processes of audio-visual integration. Brain Res 1242:136–150, 2008.

Efron B, Tibshirani R. Bootstrap methods for standard errors, confidenceintervals and other measures of statistical accuracy. Statist Sci 1: 54–77,1986.

Giard MH, Peronnet F. Auditory-visual integration during multimodal objectrecognition in humans: a behavioral and electrophysiological study. J CognNeurosci 11: 473–490, 1999.

Grefkes C, Weiss PH, Zilles K, Fink GR. Crossmodal processing of objectfeatures in human anterior intraparietal cortex: an fMRI study impliesequivalencies between humans and monkeys. Neuron 35: 173–184, 2002.

Gross J, Kujala J, Hamalainen M, Timmermann L, Schnitzler A, SalmelinR. Dynamic imaging of coherent sources: Studying neural interactions in thehuman brain. Proc Natl Acad Sci USA 98: 694–699, 2001.

Jaekl PM, Harris LR. Sounds can affect visual perception mediated primarilyby the parvocellular pathway. Vis Neurosci 26: 477–486, 2009.

Hadjipapas A, Hillebrand A, Holliday IE, Singh KD, Barnes GR. Assess-ing interactions of linear and nonlinear neuronal sources using MEGbeamformers: a proof of concept. Clin Neurophysiol 116: 1300–1313, 2005.

Hartcher-O’Brien J, Gallace A, Krings B, Koppen C, Spence C. Whenvision ‘extinguishes’ touch in neurologically-normal people: extending theColavita visual dominance effect. Exp Brain Res 186: 643–658, 2008.

Kaiser J, Hertrich I, Ackermann H, Mathiak K, Lutzenberger W. Hearinglips: gamma-band activity during audiovisual speech perception. CerebCortex 15: 646–653, 2005.

Koppen C, Alsius A, Spence C. Semantic congruency and the Colavita visualdominance effect. Exp Brain Res 184: 533–546, 2008.

Koppen C, Spence C. Assessing the role of stimulus probability on theColavita visual dominance effect. Neurosci Lett 418: 266–271, 2007a.

Koppen C, Spence C. Seeing the light: exploring the Colavita visual domi-nance effect. Exp Brain Res 180: 737–754, 2007b.

Kotter R, Wanke E. Mapping brains without coordinates. Philos Trans R SocLond B Biol Sci 360: 751–766, 2005.

Kutas M, Federmeier KD. Electrophysiology reveals semantic memory usein language comprehension. Trends Cogn Sci 4: 463–470, 2000.

Laurienti PJ, Burdette JH, Maldjian JA, Wallace MT. Enhanced multi-sensory integration in older adults. Neurobiol Aging 27: 1155–1163, 2006.

Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT. Semanticcongruence is a critical factor in multisensory behavioral performance. ExpBrain Res 158: 405–414, 2004.

Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH.Cross-modal sensory processing in the anterior cingulate and medial pre-frontal cortices. Hum Brain Mapp 19: 213–223, 2003.

Macaluso E, George N, Dolan R, Spence C, Driver J. Spatial and temporalfactors during processing of audiovisual speech: a PET study. Neuroimage21: 725–732, 2004.

Malach R, Reppas JB, Benson RR, Kwong KK, Jiang H, Kennedy WA,Tootell RB. Object-related activity revealed by functional magnetic reso-nance imaging in human occipital cortex. Proc Natl Acad Sci USA 92:8135–8139, 1995.

Maravita A, Spence C, Kennett S, Driver J. Tool-use changes multimodalspatial interactions between vision and touch in normal humans. Cognition83: B25–B34, 2002.

McDonald JJ, Teder-Salejarvi WA, Hillyard SA. Involuntary orienting tosound improves visual perception. Nature 407: 906–908, 2000.

McGurk H, MacDonald J. Hearing lips and seeing voices. Nature 264:746–748, 1976.

McIntosh AR, Bookstein FL, Haxby JV, Grady CL. Spatial pattern analysisof functional brain images using partial least squares. Neuroimage 3:143–157, 1996.

McIntosh AR, Lobaugh NJ. Partial least squares analysis of neuroimagingdata: applications and advances. Neuroimage 23, Suppl 1: S250–S263,2004.

Miller J. Divided attention: evidence for coactivation with redundant signals.Cogn Psychol 14: 247–279, 1982.

Molholm S, Ritter W, Javitt DC, Foxe JJ. Multisensory visual-auditoryobject recognition in humans: a high-density electrical mapping study.Cereb Cortex 14: 452–465, 2004.

Molholm S, Sehatpour P, Mehta AD, Shpaner M, Gomez-Ramirez M,Ortigue S, Foxe JJ. Audio-visual multisensory integration in superiorparietal lobule revealed by human intracranial recordings. J Neurophysiol96: 721–729, 2006.

Moran RJ, Molholm S, Reilly RB, Foxe JJ. Changes in effective connec-tivity of human superior parietal lobule under multisensory and unisensorystimulation. Eur J Neurosci 27: 2303–2312, 2008.

Murray MM, Camen C, Gonzalez Andino SL, Bovet P, Clarke S. Rapidbrain discrimination of sounds of objects. J Neurosci 26: 1293–1302, 2006.

Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D,Gonzalez Andino S, Schnider A. Rapid discrimination of visual andmultisensory memories revealed by electrical neuroimaging. Neuroimage21: 125–135, 2004.

Nichols TE, Holmes AP. Nonparametric permutation tests for functionalneuroimaging: a primer with examples. Hum Brain Mapp 15: 1–25, 2002.

Nishitani N, Hari R. Viewing lip forms: cortical dynamics. Neuron 36:1211–1220, 2002.

Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ. The effect ofprior visual information on recognition of speech and sounds. Cereb Cortex18: 598–609, 2008.

Noppeney U, Ostwald D, Werner S. Perceptual decisions formed by accu-mulation of audiovisual evidence in prefrontal cortex. J Neurosci 30:7434–7446, 2010.



by 10.220.32.247 on Septem


Dow

nloaded from


Peiffer AM, Mozolic JL, Hugenschmidt CE, Laurienti PJ. Age-relatedmultisensory enhancement in a simple audiovisual detection task. Neurore-port 18: 1077–1081, 2007.

Pourtois G, de Gelder B. Semantic factors influence multisensory pairing: atranscranial magnetic stimulation study. Neuroreport 13: 1567–1573, 2002.

Quraan MA, Cheyne D. Reconstruction of correlated brain activity withadaptive filters in MEG. Neuroimage 49: 2387–2400, 2010.

Robinson S, Vrba J(editors) . Functional Neuroimaging by Synthetic Aper-ture Magnetometry. Sendai, Japan: Tohoku Univ. Press, 1998.

Schneider TR, Debener S, Oostenveld R, Engel AK. Enhanced EEGgamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming. Neuroimage 42: 1244–1254, 2008.

Sekihara K, Nagarajan SS, Poeppel D, Marantz A, Miyashita Y. Recon-structing spatio-temporal activities of neural sources using an MEG vectorbeamformer technique. IEEE Trans Biomed Eng 48: 760–771, 2001.

Senkowski D, Saint-Amour D, Kelly SP, Foxe JJ. Multisensory processingof naturalistic objects in motion: a high-density electrical mapping andsource estimation study. Neuroimage 36: 877–888, 2007.

Sinnett S, Soto-Faraco S, Spence C. The co-occurrence of multisensorycompetition and facilitation. Acta Psychol (Amst) 128: 153–161, 2008.

Sinnett S, Spence C, Soto-Faraco S. Visual dominance and attention: theColavita effect revisited. Percept Psychophys 69: 673–686, 2007.

Snodgrass JG, Vanderwart M. A standardized set of 260 pictures: norms forname agreement, image agreement, familiarity, and visual complexity. J ExpPsychol Hum Learn 6: 174–215, 1980.

Stein BE, Burr D, Constantinidis C, Laurienti P, Meredith MA, PerraultTJ, Ramachandran R, Röder B, Rowland BA, Sathian K, SchroederCE, Shams L, Stanford TR, Wallace MT, Yu L, Lewkowicz DJ.

Semantic confusion regarding the development of multisensory integration:a practical solution. Eur J Neurosci 31: 1713–1720, 2010.

Talairach J, Tournoux P. Co-Planar Stereotaxic Atlas of the Human Brain,translated by Rayport M. New York: Thieme Medical, 1988.

Teder-Salejarvi WA, Di Russo F, McDonald JJ, Hillyard SA. Effects ofspatial congruity on audio-visual multimodal integration. J Cogn Neurosci17: 1396–1409, 2005.

Teder-Salejarvi WA, McDonald JJ, Di Russo F, Hillyard SA. An analysisof audio-visual crossmodal integration by means of event-related potential(ERP) recordings. Brain Res Cogn Brain Res 14: 106–114, 2002.

Thorpe S, Fize D, Marlot C. Speed of processing in the human visual system.Nature 381: 520–522, 1996.

van Atteveldt NM, Blau VC, Blomert L, Goebel R. fMR-adaptation indi-cates selectivity to audiovisual content congruency in distributed clusters inhuman superior temporal cortex. BMC Neurosci 11: 11, 2010.

Van Petten C, Luka BJ. Neural localization of semantic context effects inelectromagnetic and hemodynamic studies. Brain Lang 97: 279–293, 2006.

Van Wassenhove V, Grant KW, Poeppel D. Visual speech speeds up theneural processing of auditory speech. Proc Natl Acad Sci USA 102: 1181–1186, 2005.

Weisberg J, van Turennout M, Martin A. A neural system for learning aboutobject function. Cereb Cortex 17: 513–521, 2007.

Yuval-Greenberg S, Deouell LY. What you see is not (always) what youhear: induced gamma band responses reflect cross-modal interactions infamiliar object recognition. J Neurosci 27: 1090–1096, 2007.

Yuval-Greenberg S, Deouell LY. The dog’s meow: asymmetrical interactionin cross-modal object recognition. Exp Brain Res 193: 603–614, 2009.



by 10.220.32.247 on Septem


Dow

nloaded from


The co-occurrence of multisensory facilitation and cross ......sounds and visual objects. Behavioral...

Documents

Transcript of The co-occurrence of multisensory facilitation and cross ......sounds and visual objects. Behavioral...