Modeling the Mind: High-Field fMRI-Activation …...Topics in Magnetic Resonance. Imaging (1999),...

Topics in Magnetic Resonance. Imaging (1999), 10, 16-36 Modeling Mind and Brain

Modeling the Mind: High-Field fMRI-Activation During Cognition

Patricia A. Carpenter and Marcel Adam Just

Center for Cognitive Brain Imaging, Department of PsychologyCarnegie Mellon University, Pittsburgh, PA

There are three signal events in the conceptual history of radiology. The first was obviouslyRoentgen’s discovery of the use of x-rays to image tissue inside of the human body in 1895.The second conceptual event, which followed a short time later, was the realization that x-rayscould be used to image not only anatomy, but also dynamic physiological function inside thehuman body. The significance of the latter event, by now incorporated in the knowledge of everyeducated person, is that the human life is supported by a vast network of interlockingphysiological systems, such as the vascular and gastrointestinal systems. But in this article wewill argue that a third conceptual development in radiology will ultimately provide the deepestinsights into the nature of human life. This third event is the development of functionalneuroimaging in the 1980’s and 1990’s, initially PET-based and now involving fMRI, whichenables the imaging of the cortical functions subserving cognitive processing. Functionalneuroimaging is capable of depicting not only physiological function in the brain, but alsocognitive function. This latter meaning of “function” is the first medical and scientific opportunityto perform radiology not only on the brain, but also of the mind. In this article, we describe someof the properties of the cognitive systems that are beginning to be revealed with the use of high-field fMRI and consider some of the implications of these insights for scientific research andmedical practice.

Because the central nervous system itself is dynamic, the emerging view from fMRI is beginningto accommodate these dynamics. “Dynamic” refers to two interrelated attributes: first, that thesystem changes over time and second, that energy transfer is a key principle underlying theobserved patterns of change. For example, in sentence comprehension, dynamic cognitionconsists of information flow and computation at the level of the symbols that compose mentalrepresentations of words, phrases and meanings. In this paper, we will focus on the dynamics ofhigh-level cognition and its cortical implementation as revealed by fMRI with paradigms that testhigh-level cognitive processes. Second, we will describe some simulation models that constitutetheoretical descriptions of the dynamic properties of these systems. We begin by focusing onthe language and spatial processing systems as model systems, but we propose that theprinciples are general, as we indicate below.

The Cortical System UnderlyingLanguage Processing

Cognitive function is subserved by large-scalenetworks.

Language function emerges not from theactivation of a single brain area, but from largescale, information processing networks in thebrain. Thus, a major research challenge in thefunctional neuroimaging of cognition is not justto identify which cortical areas are involved(although that is a necessary ingredient), but

rather to determine how the multiple areas worktogether to achieve comprehension. The theorymotivating the proposed experiments is thatthere is not a one-to-one mapping betweencortical sites and comprehension functions.Rather, the system consists of interacting andhierarchically arranged subsystems that vary inthe relative contributions that they make toparticular computations and that map ontomultiple cortical locations. The diagram inFigure 1 suggests that the mapping from brainsite to behavior is not direct, but is mediated byan intermediate plane of computations that


2

constitute the functional network (see Mesulam,1990). Various computations are distributedacross multiple cortical and subcortical sites.The distribution can be viewed as weighted

(reflected in the varying thickness of the lines);one site may be a primary site for a particularcomputation, but other sites may also participateto lesser extents.

Figure 1. A figure schematically depicting the emerging conception of how cognitiveperformance, such as reading or problem solving, relate to cortical activity. Cognitiveperformance emerges from multiple levels of intermediate computations, for example,letter and word encoding, interpreting words and phrases, relating them to previousknowledge, and so forth. Each computation may map onto multiple cortical areas,although there may be a primary area that supports it, indicated by the thick lines. Inaddition, each cortical area may support multiple computations.

As a sentence like this one is being read, theactivated networks includes the cortical systemssupporting fundamental languagecomprehension processes, systems supportingthe visual processing of the written words, andsystems related to attention and the motoractions that guide visual attention and theconcomitant shift of eye fixations. Thedistribution of activation in networks of corticalareas is graphically shown in the fourteen axialoblique slices in Figure 2, going from the top of

the parietal cortex (the first, leftmost slice) downthrough the cerebellum. The figure shows thethresholded activation maps superimposed onstructural images for a gradient echo, resonantecho planar MRI with BOLD contrast at 3.0T. Itshows in white the voxels that are activatedsignificantly higher when the participant read asequence of sentences (typically, takingapproximately 60 s overall) compared to abaseline condition in which the participant wassimply fixating a point.


3

Figure 2. Statistical probability maps superimposed on structural images for a singleindividual to illustrate how a cognitive task (like sentence comprehension) elicitsactivation in multiple cortical loci. The voxels in white are those that are significantlyactivated above a baseline fixation condition when a normal college student reads aseries of sentences. The 14 axial oblique slices were acquired with gradient echo,resonant echo planar MRI at 3.0T.

Two key cortical regions that are important forlanguage comprehension are the left posteriorsuperior and middle temporal gyri (roughly,Wernicke’s area) and the left inferior frontalgyrus (Broca’s area). Note that the activation inBroca’s area and in Wernicke’s area actuallyconsists of several subregions across severalslices, and for this particular task and individual,the activation is very left lateralized. The figurealso shows activation in several areas related tovisual processing, including activation in theprimary visual areas and extrastriate cortex, in

the posterior part of the slices in third row.Activation is also found in the angular gyrus,presumably also partially related to the visualprocessing of the text. The activation that ispartially related to the shifting of visual attentionand the concomitant execution of eye fixationsduring reading is in the parietal regions,particularly the intraparietal sulcus, and thesupplementary motor region. The activation inthe precentral sulcus may be partially related tomotor aspects of reading because activation inthese regions is often found in more purely


4

motor tasks. However, its function may alsoinclude more cognitive functions, perhapsrelated to cognitive attention. Specificallyactivation in this region correlates withsuccessful memory recognition for individualwords (Buckner et al, 1998). Because thesentences (or the task) were conceptuallydemanding for this participant, activation is alsofound in the left middle frontal gyrus, labeledhere as DLPFC (dorsolateral prefrontal cortex)on the slices in the second row. While we havedescribed at least one major function associatedwith many of these regions, the specializationmay only be relative and computations areprobably distributed across multiple regions.The task is to understand how these multiplecortical regions work together in reading theprinted letters and recognizing the words,constructing the syntactic and semanticinterpretation of the phrases and clauses, andorganizing a coherent representation of thesentence’s meaning. This finding of a co-activation of a network of regions during athinking task is evident in every fMRI study oflanguage or of any other higher level cognitivefunction. It is a fundamental fact of brainfunction, which underlies all complex thinking.Although the empirical finding is notcontroversial, its interpretation raises afundamental issue in functional neuroimaging,namely, the localization of cognitive functions.

Cognitive function entails physiological work.

A cognitive system that is challenged by aharder task compared to an easier version of thesame task responds over the short term byrecruiting more neural elements within thesystem and sometimes by recruiting other,related systems. One implication of this claim isthat the localization of a function depends notjust on what the cognitive system is doing, butalso on how hard it has to work to do it. Thisconclusion and its implication contrast with anearlier assumption that the neural “hardware” isfixed. Instead, it suggests that within somebounds, the allocation is more dynamic and inpart, it is a function of the task’s demand

Early functional imaging of the languagesystem.

Some of the early resting-state studies of aphasicpatients provided some indications of both the

network properties of the language system andits workload sensitivity. The network-relatedresult, from several FDG-based PET studies,was that over 96% of the many tens of aphasicpatients who were tested had a common site ofmetabolic impairment, a hypometabolism, in theleft temporal and temporoparietal regions,measured when the patient was at rest and notperforming any task (Karbe et al 1989; Kempleret al 1991; Metter et al.1990). It is remarkablethat this generalization held regardless of the siteof the structural lesion, including subcorticalsites, and regardless of clinical categories ofaphasia. The result that any aphasia-inducingdamage ultimately changes the function of acommon site indicates that a variety of regionsparticipate in a network function that includesthis common site.

Another fascinating aspect of these early PETresults was that the degree of an aphasicpatient’s metabolic impairment (defined as thedegree of hypometabolism) substantiallycorrelated with the degree of impairment insentence comprehension (measured off-line bystandardized tests). For example, in one of thesestudies (Kempler et al 1991), the resting statecerebral glucose metabolic rate (which washypometabolic) in the left temporoparietalregions (but not others) correlated withcomprehension performance in a common test ofaphasic language skill, the Western AphasiaBattery (WAB) (r = .44 for the laterosuperiortemporal gyrus and r = .60 for the middletemporal gyrus, areas that include Wernicke’sarea.). This result speaks not only to thenetwork properties of the language system, butalso to the resource properties of the network.Specifically, the measure of resting PET activitymay provide an index of the size of the patients'potential resource supply, which could place anupper limit on sentence comprehension. Thisinterpretation is the basis of a computationalmodel of aphasic sentence comprehension thatwe have developed with Henk Haarmann(Haarmann et al, 1997). The model can producea semantic interpretation of a sentence andanswer questions about the content of thesentence. The model’s information processingfunctions consume resources, and the model’sperformance degrades as its resource pool isdecreased. With inadequate resources, the modelproduces a partial representation of a sentence,the more so if the sentence is demanding. The


5

model’s degradation in comprehensionperformance with decreasing resources and withincreasing sentence complexity provides a goodmatch to the comprehension performance ofaphasic patients who vary in the severity of theirimpairment. This earlier model was not a modelof cortical function, because it treated the entirelanguage system as a single entity rather than anetwork of collaborating areas. Nevertheless, themodel captured the resource-sensitive nature oflanguage processing, and it appropriatelycharacterized the impact of differential severityof brain damage as differential resourcedepletion.

An fMRI study of task demand in languagesystem.

In an initial demonstration of the cognitiveworkload effect in the domain of languagecomprehension, we found that as sentencecomprehension becomes more difficult becauseof syntactic and semantic features, there is anincrease in the fMRI-measured activation in anetwork of cortical regions. These regionsinclude the left perisylvian cortical regions thatare classically associated with languageprocessing and regions in the right homologue(Just, Carpenter, Keller, Eddy & Thulborn,1996).

The amount of activation in four cortical areas(Wernicke’s, Broca’s, and their right hemispherehomologues) was examined in 15 normal right-handed young adults, as a function of thedemand imposed by the comprehension of threesentence types. The sentences are superficiallysimilar (each containing two clauses and thesame number of content words) but they differin structural complexity, and hence, in thedemand they impose in the sense of how muchprocessing is needed to figure out who is doingwhat to whom. To illustrate the surface structuresimilarities among the three sentence types,we’ve used the same words. However, in thestudy, each sentence had different words thathad been randomly assigned to differentsentence types.

1. ACTIVE CONJOINEDThe reporter attacked the senator and admitted the error.

2. SUBJECT RELATIVE CLAUSEThe reporter that attacked the senator admitted the error.

3. OBJECT RELATIVE CLAUSEThe reporter that the senator attacked admitted the error.

Whereas type 1 sentences contain active clausesthat are simply conjoined, the more complextype 2 sentences contain a relative clause thatinterrupts a main clause, causing additionalmaintenance. Finally, in type 3 sentences themain clause is interrupted, and the first nounplays different roles in the two clauses (as thesubject of the main clause and the object of therelative clause). The most complex type (type 3,Object Relatives) produces longer processingtimes, higher error rates, and larger increases inpupil dilation (another measure of cognitiveeffort) than the less complex type (type 2,Subject Relatives) (Just & Carpenter, 1993;King & Just, 1991)

The experiment involved five differentconditions. In conditions of type 1, 2, or 3, theparticipant read several different sentences ofthe appropriate type successively, and after eachsentence, answered a visually presented questionsuch as, “Who did the attacking?” In a fourthcondition, intended to assess visual processing,the participant was shown a series of “nonsense”words made of consonant strings that they wereto scan. In the fifth condition, the participantsimply fixated an asterisk, and the activationduring this condition provided a baseline againstwhich all the other conditions were compared.On average, approximately 60 images weretaken during each condition for each participant.In all of the analyses, the average activationlevel for each voxel in a region of interest (ROI)during the experimental conditions wasstatistically compared to the activation duringthe baseline conditions using a t-statistic toidentify which voxels were significantlyactivated.

This earlier study of ours was done at 1.5 Tesla,using gradient echo, resonant echo planar MRI,with BOLD contrast. The acquisition parameterswere TR = 1500 ms, TE = 50 ms, flip angle =90°, voxel size = 3.125 x 3.125 x 5 mm, 7 axial


6

slice planes, slice thickness = 5mm, 1mm gap,acquisition matrix =128 x 64, FOV = 400 x 200mm. The images from more recent studies wereobtained at 3.0T, changing the TR to 3000 msand the TE to 25 ms. At both field strengths, TEwas chosen to match T2

* to ensure maximumsensitivity of BOLD contrast. The larger TR at3.0T allows more slices to be acquired to covermore of the brain.

Because our approach focuses on measuring theamount of activity in some area, it is importantto have an a priori definition of a cortical area.That is, we ask the question “how is theactivation modulated by task demand,” so wemust know where to look for the activation. Bycontrast, other approaches that focus on the“where” question (what are the locations of theactivation in such and such task) need onlydetermine the centroids of activations, oftendone with respect to Talairach coordinates. Ourapproach makes use of anatomically definedregions of interest (ROI’s), which are drawn foreach individual subject. It would be preferable touse a universal parcellation scheme that is basednot only on structural properties of the brain butalso on known functional (activation) propertiesas well, but unfortunately no such scheme yetexists. We use the anatomical parcellationscheme developed by Rademacher et al. (1992),one view of which is shown in Figure 3. Thisparcellation uses limiting sulci and anatomicallylandmarked coronal planes to segment corticalregions. To co-register the functional andstructural images, a mean of the functionalimages is co-registered to a high-resolutionstructural volumetric scan (SPGR), and limitingsulci are identified by viewing the structuralimages simultaneously in the three orthogonalplanes. The functional images are thensegmented in the functional acquisition plane bymanually tracing the regions of interest on eachslice. To facilitate this tracing, we start with atemplate of the ROI that has been defined in atemplate brain, and then we transform thosetemplates to fit each subject’s brain, adjustingthem according to each subject’s structurallandmarks. In the nomenclature of Rademacheret al. (1992), the area we refer to as Wernicke’sarea includes areas T1p, T2p, TO2, PT andportions of areas SGp and AG (Brodmann’sareas 22, 37, 39, 40 and 42). (Area PT, planumtemporale, is not visible in this figure). Figure 4indicates a slice through part of this area in

sagittal view for one participant. The area werefer to as Broca’s area corresponds to areasF3o, F3t, FOC and FO (Brodmann’s areas 4, 6,44, 45, and 47). (Area FO is not visible in thisfigure). The dorsolateral prefrontal regioncorresponds to area F2 (Brodmann’s Areas 8, 9,and 46).

Figudevprovregimeaareacomceremorby RCavNeu

Figustudmos

re 3. The anatomical parcellation schemeeloped by Rademacher et al. (1992)ides a convenient way to refer to cortical

ons when quantitatively assessing fMRI-sured activation. This view highlights an of key interest in our studies of languageprehension. (Adapted from “Humanbral cortex: Localization, parcellation, andphometry with magnetic resonance imaging”ademacher, Galaburda, Kennedy, Filipek, &

iness, 1992. Journal of Cognitiveroscience, 4, Figure 1, p. 354.)

re 4. The sagittal scouts for a collegeent, indicating the axial slice that elicited thet activation in the posterior superior


7

temporal gyrus in a reading comprehensionstudy.

Figure 5 shows the thresholded fMRI activationimages (1.5T) superimposed on structuralimages for the most activated slice for oneparticipant. Each image shows the results forone condition for this slice, showing in white thevoxels that are significantly more activated thanthe resting baseline, using a t-test to compare theaverage activation level of each voxel in eachsentence condition to its level during restepochs. The structural image of the most activeslice through Wernicke’s area (indicated by thebox). This particular slice shows little of theactivation in the right homologue or in Broca’sarea or its right-hemisphere homologue. Theincreasing number of white voxels in the boxillustrates how the number of significantly

activated voxels increases as the complexity ofthe demand increases, that is, in going fromconsonant strings to simple conjoined activesentences, to sentences with embedded subjectrelative clauses and finally, to sentences withembedded object relative clauses. Because theactivation in Broca’s area was difficult toevaluate precisely given the location of the axialscans, the activation was assessed in five of the15 participants (hence, the larger error bars) whowere additionally scanned in an ancillary studyusing coronal slices and presenting comparablesentences. While this shows the effect for onlyone participant, the results were supported bythe analysis of the volume of activation over thethree most relevant slices across all of theparticipants.

Figure 5. Thresholded fMRI activation images (1.5T) (superimposed on structuralimages) for only the most activated slice through Wernicke’s area (indicated by thearrow) from one participant. The number of activated voxels (shown in white) generallyincreases with sentence complexity. (From “Brain activation modulated by sentencecomprehension” by Just, Carpenter, Keller, Eddy, & Thulborn, 1996, Science, 274, Figure2, p. 115. Copyright 1996 by the American Association for the Advancement of Science.Reprinted with permission).

The results, after being aggregated over all theparticipants (as shown in Figure 6), indicate thatthe processing of more complex sentences leadsto an increase in the volume of neural tissue thatis highly activated in all four areas: Wernicke’sarea, Broca’s area, and their right hemispherehomologues. We use the terms “Wernicke’sarea” and “Broca’s area,” but defined withrespect to Rademacher’s scheme,acknowledging that there are no clear cyto-

architectonic boundaries. The increasedactivation also occurred in a second measure, thepercentage of activation compared to the level inthe fixation condition. The importance of theincrease in activation is that it suggests that thelanguage comprehension system responds toincreased demand by increasing the areas thatare involved (activation of the right hemispherehomologues) and increasing the contribution ofactivity within that region or adjacent regions.


8

Figure 6. The average number of activated voxels across participants indicates that theprocessing of more complex sentences leads to an increase in the volume of neuraltissue that is highly activated in all four areas. The top panels indicate the averagenumber of activated voxels in the left (Wernicke’s area) and right laterosuperior temporalcortex (and standard errors of the means over 15 participants). The bottom panelsindicate the average number of activated voxels in the left (Broca’s area) and right inferiorfrontal cortex (and standard errors of the means over only five participants). (From “Brainactivation modulated by sentence comprehension” by Just, Carpenter, Keller, Eddy, &Thulborn, 1996, Science, 274, Figure 1, p. 115. Copyright 1996 by the AmericanAssociation for the Advancement of Science. Reprinted with permission).

A graphic display of the increase in activationwith sentence demand.

The contour plots shown in Figure 7simultaneously display topographic informationand amplitude information. The planecorresponds to brain topography and the height

of the points corresponds to the amplitude of thevoxels’ activation increases over baseline levels(actually, their t values, a proxy for theiramplitude that controls for variance). The left-hand plots show the voxels’ t values in the active(conjoined) condition, and the right hand plotsshow the most demanding object-relative


9

condition, for the axial slice with the mostactivation. The three pairs of plots are data fromthree participants

Figure 7. Contour plots that simultaneouslydisplay topographic information and amplitudeinformation for the easier conjoined activesentences (left hand plots) and the harder objectrelative sentences (right hand plots). The planecorresponds to brain topography and the heightof the points corresponds to the amplitude of thevoxels’ activation increases over baseline levels(actually, their t values, a proxy for theiramplitude that controls for variance). The threepairs of plots are data from three participants.(From “Brain activation modulated by sentencecomprehension” by Just, Carpenter, Keller,Eddy, & Thulborn, 1996, Science, 274, Figure 3,p. 115. Copyright 1996 by the AmericanAssociation for the Advancement of Science.Reprinted with permission).

The mountain range of voxels that is activated inthe active-conjoined condition appears to eruptfurther still in the object relative condition. Thenew voxels that become activated in the objectrelative condition tend to fill in and augmentexisting groups of activated voxels. The non-participating voxels are doing relatively little inboth of these sentence conditions. So the effectof increasing the sentence demand is twofold.

First, the increase in demand increases thenumber of activated voxels (by pushing tothreshold voxels in the foothills and intersticesof the mountain range that had been onlypartially activated in the less demandingcondition). This effect occurs in a very largemajority of the participants. Second, the increasein demand increases the activation level of someof the previously activated voxels. The lattereffect occurs for a majority of the participants,but not a large majority. Thus, making the brainwork harder on sentence comprehensionincreases the volume and the magnitude of itsactivation. Finally, these plots show that some ofthe activated voxels are spatially contiguous ornearby.

The contour plots suggest that as demandincreases, adjacent and interstitial voxels thatwere near the threshold level now rise above thethreshold level, and the already activated voxelsincrease their activation level. One interpretationof the increase in the volume of activation issimply that the increased demand causes arecruitment of additional neuronal tissue.Another interpretation is that some subvolumeof the additionally recruited voxels had beenactivated even in the condition with the easiestsentences, but that the total activation in theentire voxel was not enough to bring it to thethreshold level. Both interpretations entail thatthere is more brain activation for more difficultsentences.

The precise functions of the four regions are notdelineated by this task, except to suggest that theadditional processing is not lexical (given thatlexical content was equivalent acrossconditions). The modulated activation of theright homologue of Wernicke’s area isconsistent with the proposal that it, too, is part ofthe language network, in addition to beingevoked in the processing of figurative (non-literal) language (Bottini et al., 1994) andprosodic information (Tompkins & Mateer,1985). Thus, language comprehension, likeother cognitive attainments, is accomplished bya network that spans several cortical regionsacross both hemispheres, as well as subcorticalregions that are not here the focus. Moregenerally, this study demonstrates that the neuralsystem activation does not reflect simply thequalitative nature of the demand, but also theamount of demand. To meet the increased


10

demand as the difficulty of the task increases,the system recruits more regions.

Reading compared to listeningcomprehension.

One reason to believe that the left posteriorsuperior and middle temporal regions and theinferior frontal regions are involved infundamental language comprehension processesis because these regions consistently showactivation both in reading comprehension and inlistening comprehension. To make thiscomparison, the task is kept the same as wedescribed earlier, but instead of presenting thesentences in written form, they are presentedauditorily, and the data for both conditions areacquired at 3.0T. Again, the activation in boththe reading conditions and the listeningconditions are compared to the activation ofcommon baseline condition, in which theparticipant is simply fixating a point (Carpenter,Just & Keller, 1999). The patterns of activationin the two conditions show both commonalitiesand differences that can be illustrated with thedata from one individual. Figure 8 shows twoslices in an oblique axial orientation that includethe middle and superior temporal regions for thereading condition (on the left) and the listeningcondition (on the right). The reading conditionshows primarily left lateralized activation in theposterior superior and middle temporal regions,with only a little bilateral activation in the moresuperior slice, as well as activation in the moreposterior visual regions. The listening conditionshows no activation in the primary visualregions, but considerable bilateral activation,presumably partly associated with the earlyauditory processing. In addition, notice thatthere is considerable overlap in the patterns ofactivation, particularly in the left posteriormiddle temporal region, presumably due to theshared language interpretation processessupported by this neural region. The partialoverlap suggests that computations subserved bythis region are evoked in both comprehensiontasks, irrespective of the modality.

Figure 8. Two oblique axial slices (the moresuperior is the upper slice) that include part ofthe middle and superior temporal regionsshowing the activation in the reading (on the left)and the listening condition (on the right) for asingle individual. The reading condition showsprimarily left lateralized activation in theposterior superior and middle temporal regions,with a little bilateral activation in the moresuperior slice, as well as activation in the moreposterior visual regions in both slices. Thelistening condition shows no activation in thevisual regions, but considerable bilateraltemporal activation, associated with the auditoryprocessing. In addition, the overlap in thepatterns of activation, particularly in the leftposterior middle temporal region and to a lesserextent right temporal region, are presumably dueto the language interpretation processessupported by these regions.

Spatial resolution.

A fascinating aspect of high-field fMRI is that itreveals that cortical neural systems areorganized at several different spatial grain sizes.While most of our studies have used a spatialresolution producing voxels that areapproximately 48 mm3 in volume (3.1 x 3.1 x 5

Reading Listening


mm3, this has become apparent in a few casestudies done at a much higher spatial resolution,producing voxels that are approximately 0.8 x1.6 x 3 mm3 in volume (Thulborn, Chang, Shen& Voyvodic, 1997). These smaller voxels beginto approach the magnitude of a cortical column.The high-resolution studies are difficult toperform because they can tolerate less headmotion than many participants canaccommodate. Furthermore, the additionalspatial resolution is bought at the expense ofobtaining less coverage of the brain. However, atthe high spatial resolution fMRI produces anadditional brilliant level of detail, whileretaining all the information produced by thelower resolution image. In collaboration withTim Keller, we obtained a pair of images thatcontrasts higher and lower spatial resolution bystudying the same person on the same sentencecomprehension paradigm. Figure 9 shows thatthe high-resolution image of the activation in leftsuperior temporal gyrus precisely follows themargin of the sulcus, whereas this level of detailis unavailable in the lower resolution image.

the right) shows that the activation in leftsuperior temporal gyrus precisely follows themargin of the sulcus, whereas this level of detailis unavailable in the lower resolution image.

One interpretation of the finer spatial resolutionis that it is closer to the “truth” in the sense thatthe resolution may be closer to capturing thefunctional units of neural computation.However, an alternative view is thatcomputation is occurring simultaneously atseveral levels, levels from small neural clusterswithin circumscribed regions of the cortex, onup to the large scale computations that emergefrom the coordinated activities across multiplecortical regions. If this latter perspective is amore accurate one, then there is no ideal spatialresolution and ultimately, the theoretical accountof cognition will need to span across multiplespatial as well as temporal grain sizes to accountfor how cognition unfolds in the brain.

Clinical application.

We have had the opportunity to apply this fMRI

11

Figure 9. A comparison of the activation in thesame slice obtained by studying the sameperson on the same sentence comprehensionparadigm, but at different spatial resolutions.The cut-out from the high-resolution image (on

paradigm to a surgical candidate, a patient withan arterio-venous malformation (AVM) in theleft frontal cortex. The patient was a 38-year-old woman who presented with periods of word-finding difficulties and intense headaches. Thepatient’s comprehension accuracy was well inthe normal range. Figure 10 depicts theactivation results for the patient with the AVMfrom a study done in collaboration with TimKeller and Keith Thulborn, along with thecomparable data from a college student. For thecollege student, two of the main cortical areasare activated during sentence comprehension arein the left hemisphere (on the right of theimage), reflecting the largely left-lateralizationof language processing. The cluster at the top(anterior) is the left inferior frontal gyrus(Broca’s) and the cluster at the bottom is theposterior superior temporal gyrus (Wernicke’sarea).

The image on the left from the patientdemonstrates that this fMRI paradigm providesinterpretable data concerning an abnormallanguage system. The AVM is visible as a largedarkened region in the left frontal region. Theimages show normal activation in the lefttemporal region (Wernicke’s), but an abnormallack of response in the left inferior frontal cortex

48 mm3 Voxels High Resolution


12

(Broca’s area), presumably due to the AVM.Note that this image shows there is strongactivation in the right inferior frontal cortex.While we have observed activation in the righthomologue of Broca’s area, such a stronglycontralateral pattern of activation (right frontaland left temporal) is one that we have notobserved in the testing of approximately 100normal individuals, and it suggests that more ofthe language functions have been taken over by

the right hemisphere than in the typical collegestudent. Thus, the normal language network hasbeen disrupted by the AVM, and the networkfunction has been reconstituted with the righthomologue of Broca’s area playing a larger rolein place of the damaged area. The more generalconclusion is that the disruption of a networkunderlying cognition to which a patient adapts isproduced by a network adaptation.

Figure 10. The activation results for the patient with the AVM (visible as a large darkenedarea in the left frontal region), along with the comparable data from a normal collegestudent, both performing a language comprehension task. For the college student, thetwo activated cortical areas (superior/middle temporal gyrus and inferior frontal gyrus) areboth largely left lateralized. For the patient with an AVM, the activation is cross-lateralized.

More recently, we applied a similar paradigm ina 45 year-old patient studied acutely and as hewas recovering from stroke (Thulborn,Carpenter, & Just, in press). His structural lesionwas approximately in a similar left frontal area.He presented with a dense expressive aphasia attime of stroke, which resolved into a mildanomia by 6 months. The fMRI in sentencecomprehension was done at 3 days and again at6 months post onset. We observed spontaneous

changes in brain function in the months afterstroke-induced aphasia. Specifically, there was arightward shift of language-related brainactivation to right hemisphere homologues ofdamaged and undamaged areas. In other words,there was a readjustment of the functionalnetwork that was adaptive to the physicaldamage during a period of recovery of languagecomprehension.

-----Area of malformation


comprehension apply equally to the visuo-spatialprocessing system. The spatial processingparticipates in various rigid and non-rigid spatialtransformations that are evoked when reasoning

13

Figure 11. The activation results in the sentencereading task for a patient 6 months post strokeonset, after considerable spontaneous languagerecovery. The pattern suggests a rightward shiftof language-related brain activation to righthemisphere homologues of damaged andundamaged areas. This shift is shown here bythe circles around some of the activation in theright homologue of Broca’s and Wernicke’sareas. (Adapted from “Plasticity of language-related brain function during recovery fromstroke” by Thulborn, Carpenter, & Just, Stroke,in press.)

The final pattern of activation during sentencecomprehension is somewhat similar for thestroke patient and the AVM patients, althoughthey came to be in this state through routes thatoccurred at very different times of their lives.We can speculate that in the AVM patient, theAVM may have preceded the development ofthe language function in childhood, and that theresulting cross-lateralization reflects theplasticity of brain function during earlychildhood. The network adaptivity seems similardespite the differences in age of onset.

The Cortical Systems Underlying Visuo-spatial Processing

The principles that we have described in thecontext of the cortical system for language

about objects in space, as for example, when aneuro-radiologist imagines how a series of twodimensional MR images might be mentallycombined and mapped onto a three dimensionalbrain. Numerous behavioral studies havesuggested that in young adults, the systemssupporting language comprehension andvisuospatial reasoning are somewhat separable(Carroll, 1993; Jurden, 1995; Shah & Miyake,1996), and so it is perhaps not surprising thatthere is some partial dissociation cortically aswell.

The cortical system supporting visuo-spatialprocessing has several specialized subsystems,but we focus on two main pathways thatoriginate in the primary visual area and thenproject forward with extensive connections inthe visual processing areas, including V2, V4and V5. One pathway, the ventral stream, feedsinto the inferior temporal lobe and is largelyspecialized for object recognition (Farah, 1990;Ungerleider & Haxby, 1994). The secondsubsystem, the dorsal stream, connects to theparietal lobe and is involved in spatial analysisand processing (Ungerleider & Mishkin, 1982;Ungerleider & Haxby, 1994). A third focus ofinterest includes the motor systems that computeeye movements in response to internal switchesof attention, and which includes the precentralgyrus and the posterior middle frontal gyrus, andother cortical areas around and along theinterhemispheric fissure.

These regions are co-activated in the visuo-spatial task of mental rotation. In this paradigm,a participant judges whether two pictures depicteither the same object (but possibly at differentorientations), as illustrated by the top pair ofthree-dimensional cube figures in the top ofFigure 12 (Arnoult, 1954; Shepard & Metzler,1971). On some trials, the participant is showntwo mirror image isomers, as in the middle rowof Figure 12. Participants report that theyimagine one figure rotating into the orientationof the other figure; consistent with these selfreports, the average decision time increasesmonotonically with the angular disparitybetween two pictures of the same object(Shepard & Metzler, 1971). The underlying


14

cognitive processes involve mentallyrepresenting the objects, imaging part of one ofthem at successive orientations that close thedistance between the objects’ orientations, andthen evaluating whether the rest of the objectlines up with the target object (Just & Carpenter,1976)

Figure 12. Stimuli from a mental rotation task,adapted from Shepard and Metzler (1971). Theparticipant must judge whether a pair of figures,such as the ones at the top or those in themiddle, represent the same object or mirrorimage isomers. The bottom shows two gridsthat participants were asked to scan, square bysquare, to compare the activation associatedwith eye fixations to that associated with mentalrotation. (From “Graded functional activation inthe visuo-spatial system with the amount of taskdemand” by Carpenter, Just, Keller, Eddy, &Thulborn, 1999, Journal of CognitiveNeuroscience, Figure 1, p.10. Copyright 1999 bythe MIT Press. Reprinted with permission).

Earlier neuroimaging studies showed activationin the parietal region during mental rotation(Cohen et al., 1996; Alivisatos & Petrides,1997), supporting the hypothesis derived fromsingle-cell studies that this cortical region ispartially involved in computing visuo-spatialcoordinates at successive orientations. Based onthe resource demand perspective, wehypothesized that there should be an increase infMRI-measured activation in the parietal regionsas a function of the amount of mental rotation(Carpenter, Just, Keller, Eddy & Thulborn,

1999a). The experimental conditions involved agraded manipulation of the angular disparitybetween the two figures, either 0°, 40°, 80° or120°, along with a baseline fixation condition.Based on a computational model of the task, wepredicted a monotonic increase in parietalactivation as a function of the increase inangular disparity. Specifically, larger angulardisparities require more resources for bothcomputing more intermediate orientations andfor maintaining representations of both stimulibeing compared. Finally, to contrast taskdemand with simply task duration, we includeda condition in which participants visuallyscanned a fixed grid to assess the impact ofmultiple eye fixations in the absence of rotation,as shown in the bottom row of Figure 12. Thegrid-scanning task was designed to take longerthan the rotation conditions, but in spite of this,it was predicted that it should result in lessactivation than the rotation task because itinvolved very little computation.

An initial study, using a 1.5 Tesla GE scannerand 7-9 coronal slices, focused almostexclusively on activation in the parietal region.The second study used the same paradigm with a3.0T scanner and 14 slices in an axialorientation, which enabled us to quantify theactivation in more cortical regions, includingmost of the inferior temporal region and thefrontal regions. These two studies, using theidentical task paradigm, permit the evaluation ofthe effect of field strength on the results.

Figure 13 shows the positions of three slices thatillustrate the type of data we obtained. Theindicated superior slice shows some of theactivation associated with the attentional andmotor systems. The medial slice showsactivation around the intraparietal sulcusassociated primarily with the visuo-spatialcomputations, and the indicated more inferiorslice shows the activation associated with theinferior temporal regions. In Figure 14,structural images of these three slices constitutethe rows. Superimposed on the structural imagesis the average activation in each of the fiveconditions, the four rotation conditions (0°, 40°,80°, and 120°) and the grid condition, for atypical participant. The white areas indicate thevoxels that were activated significantly abovethe baseline fixation condition. The first row in


15

Figure 14 (slice 2 from the top in Figure 13) isthrough the centrum semiovale. The activation isin the area of the precentral sulcus (the frontaleye fields) and along the cortex of theinterhemispheric fissure (supplementary eyefield), and it tends to be high and similar in thegrid and four rotation conditions. The secondrow, through the cingulate gyrus, showsactivation in the intraparietal sulcus and gyri.The number of activated voxels is relatively lowin the grid condition and much higher in therotation conditions, where it tends to increasewith angular disparity. The third row, throughthe inferior temporal-occipital lobes, showsactivation to be high in the rotation conditionsand lower in the grid condition. This generaldescription of the results was supported by theanalyses of multi-slice ROI’s across theparticipants.

Figure 13. The positions of three slices in asagittal scout to illustrate the data obtained fromthree main regions of interest for one participant.The top arrow indicates a slice includes areasassociated with the attentional and motorsystems. The middle arrow indicates a slice thatincludes regions around the intraparietal sulcus,regions associated primarily with the visuo-spatial computations. The more inferior sliceincludes the inferior temporal regions that aremore associated with pattern recognition (From“Graded functional activation in the visuo-spatialsystem with the amount of task demand” byCarpenter, Just, Keller, Eddy, & Thulborn, 1999,Journal of Cognitive Neuroscience, Figure 3, p.13. Copyright 1999 by the MIT Press. Reprintedwith permission).

Figure 14. Statistical probability mapssuperimposed on structural images for the fourrotation conditions and the grid conditions for thethree slices indicated in Figure 13. The whitevoxels were activated significantly above thebaseline fixation condition. The first row (toparrow in Figure 13) is through the centrumsemiovale. The activation is in the area of theprecentral sulcus (the frontal eye fields) andalong the cortex of the interhemispheric fissure(supplementary eye field), and it tends to behigh and similar in the grid and four rotationconditions. The second row, through thecingulate gyrus, shows activation in theintraparietal sulcus and gyri. The number ofactivated voxels is relatively low in the gridcondition and much higher in the rotationconditions, where it tends to increase withangular disparity. The third row, through theinferior temporal-occipital lobes, showsactivation to be high in the rotation conditionsand lower in the grid condition. (From “Gradedfunctional activation in the visuo-spatial systemwith the amount of task demand” by Carpenter,Just, Keller, Eddy, & Thulborn, 1999, Journal ofCognitive Neuroscience, Figure 4, p. 14.Copyright 1999 by the MIT Press. Reprintedwith permission).

The major prediction concerned the effect ofrotation on activation in the parietal region,specifically whether larger angular disparities,which consume more activation resources in thecomputational model, are associated withincreased fMRI-measured activation. As Figure15 indicates, the number of significantlyactivated voxels in the parietal region, most ofwhich were in and around the region of the


16

intraparietal sulcus and into the transverseoccipital sulcus, increased monotonically withangular disparity. These data demonstrate twoimportant properties. First, the linear trendshows the quantitative impact of the amount of aparticular type of task demand on activation asassessed with fMRI, which constitutes majorsupport for the approach. Second, the bilateralityof the effects indicates the involvement of bothhemispheres in this visuo-spatial task, in contrastto the rather strong laterality of many languagecomprehension tasks. The bilaterality is mostasymmetric at 0°, where the right hemisphere is

noticeably more responsive than the left,suggesting that bilaterality may increase with thetask’s demand, as it does in language processing.The right-hand side of Figure 15 indicates thatthe increased demand associated with moremental rotation also led to an increase in theaverage percentage of activation increase overthe baseline fixation condition. Thus, as withthe sentence comprehension, the corticalsystems subserving visuo-spatial imaging showincreased activation with demand.

Figure 15. The average number of activated voxels (left hand side) and the averagepercentage of activation (right hand side) in the parietal regions of 7 subjects bothincrease monotonically with the amount of mental rotation, and both measures are higherthan for the grid condition. These data were acquired with a 3.0T scanner. (From“Graded functional activation in the visuo-spatial system with the amount of task demand”by Carpenter, Just, Keller, Eddy, & Thulborn, 1999, Journal of Cognitive Neuroscience,Figure 5, p. 15. Copyright 1999 by the MIT Press. Reprinted with permission).


17

Compared to the 3.0T results, the 1.5T studyshowed the same quantitative trends, but a muchlower number of voxels that were significantlyactivated. Figure 16 shows a monotonic increasein the number of voxels significantly activatedabove baseline in the parietal regions for the1.5T study, which involved the same paradigmand approximately the same number of subjects.This difference is consistent with the highersignal-to-noise ratio (by a factor of two[Thulborn et al., 1996]) and with the highersensitivity for magnetic susceptibility effects ofthe 3.0T compared to the 1.5T systems. In fact,the increased susceptibility of the 3.0T(approaching a quadratic power [Thulborn et al.,1982]) increases the sensitivity to themicrovasculature that biases toward the smallervessels. Both the higher signal-to-noise ratio andincreased sensitivity would enable one to detectsmaller increases in activation, which wouldincrease the number of voxels that would befound to be significantly activated.

Figure 16. The average number of activatedvoxels in the parietal regions increasesmonotonically with the amount of mentalrotation. These data, acquired with a 1.5Tscanner, show less of an effect than the dataacquired with a 3.0T scanner, shown in Figure17. (Adapted from “Graded functional activationin the visuo-spatial system with the amount oftask demand” by Carpenter, Just, Keller, Eddy,

& Thulborn, 1999, Journal of CognitiveNeuroscience, Figure 6, p. 15).

As expected, the increase in demand not onlyaffected the activation in the parietal region, butalso in the inferior temporal region, a region thatis primarily (but not exclusively) involved inobject recognition. The results, averaged acrossparticipants, are shown in Figure 17 for the 3.0Tstudy. The figure indicates a considerableelevation in the number of activated voxels andthe amplitude of activation in all four of therotation conditions, particularly compared to thegrid scanning condition. The increase is notmonotonic, indicating that the computations inthis region are more uniform throughout therotation task. What the high activation in thisregion along with the activation in the parietalregions indicates is that the mental rotation taskemerges from an interaction among systems thatscale different regions and hemispheres.

Figure 17. The average number of activatedvoxels in the inferior temporal regions iselevated for the rotation conditions compared tothe grid scanning conditions, although theincrease is not monotonic. These data wereacquired with a 3.0T scanner. (Adapted from“Graded functional activation in the visuo-spatialsystem with the amount of task demand” by


18

Carpenter, Just, Keller, Eddy, & Thulborn, 1999,Journal of Cognitive Neuroscience, Figure 7, p.17).

Communication between linguistic and visuo-spatial network.

The language and visuo-spatial networks, whichappear to be cortically segregated across theprevious studies we have reviewed, also directlycommunicate with each other. Suchcommunication suggests that the modularity ofvarious large-scale cortical networks is only oneof degree, and therefore, the cognitive systemsand the cortical neural systems that support themmay more appropriately be characterized bydegrees of interaction. We examined theinteraction between the language and visuo-spatial systems by monitoring the time course ofactivation in the key cortical regions associatedwith each system while a participant read asentence that referred to a spatial configuration.The task involves reading a sentence, such as Itis (isn’t) true that the star is above the plus, andthen verifying it against a picture, such as a starabove a plus (Just, Carpenter, Keller, Eddy &Thulborn, 1996; Carpenter, Just, Keller, Eddy &Thulborn, 1999b). Moreover, the study used anevent-related method to examine the time courseof activation at various points during thesentence processing phase and the picture-processing phase, rather than relying onasymptotic activation that sums across differentphases of the task. The study manipulated thedifficulty of the comprehension task bycomparing the comprehension of negativesentences to that for their affirmativecounterparts. Numerous behavioral studies haveindicated that negative sentences are moredifficult to process than affirmatives, resulting inincreased reading time and errors (Chase &Clark, 1972; Carpenter & Just, 1975). Thiscomplexity leads to the prediction of higherlevels of activation during the processing ofnegative sentences than affirmative sentences inthe left posterior superior temporal region,which we have shown is associated withlanguage processing. The study also examinedthe activation in the parietal regions because oftheir association with various visuo-spatialprocesses, such as mental rotation and covertspatial attention (Carpenter, Just, Keller, Eddy,& Thulborn, 1999a; Alivisatos & Petrides, 1997;Cohen et al., 1996; Lynch, Mountcastle, Talbo,

& Yin, 1977; Tagaris et al., 1997). The reasonfor examining activation in the parietal regionsis that during sentence reading, the participant ispreparing to compare the linguisticrepresentation to a picture. If the neuralprocessing of language is not encapsulated butalso involves some contact with the systemrepresenting the sentence’s referent, then theremight also be activation in the regions associatedwith visuo-spatial relations. Such an effect(which describes the obtained results) isconsistent with the hypothesis that languageprocessing is not restricted to the classicallyassociated cortical regions. The co-activation intwo regions argues against the view that theprocessing in the linguistic region is temporallyisolated from processing in other corticalregions.

A typical trial involved a sentence (such as, Itisn’t true that the star is above the plus), thatwas either affirmative or negative, followed by apicture of a star and a plus accompanied by adiagrammatic instruction to mentally rotate thearray by 0o or 135o before comparing it to thesentence. These two conditions (affirmativeversus negative sentence and 0o or 135o rotation)were varied orthogonally to manipulate thecognitive demand in each stage. To allow theactivation to return to baseline between trials, a20 s fixation epoch occurred between each trial,and the activation during the last 14 s averagedacross trials constituted the baseline. Imagingwas performed at 1.5T using gradient-echo EPI(Acquisition parameters were 7 coronal slices,TR=1500 ms, TE= 50ms, flip angle=90o, 128 x64 acquisition matrix, FOV = x cm2, 5 mmslice thickness, 1mm gap).

To monitor the time course of activation, thesentence presentation and the picturepresentation were synchronized electronicallywith the scanner. The TR of 1500 ms with 7coronal slices allowed frequent sampling of thetwo main cortical regions of interest. A total of5 slices defined the posterior temporal regions,and they were acquired with a mean elapsedtime of 940 ms for the first acquisition interval,2440 ms for the second, and so forth. For the 5slices that defined the parietal regions, thecorresponding mean elapsed time was 850 msand 2350 ms for the first and second acquisitionintervals, and so forth. The voxels of interestwere identified by first computing separate


19

voxel-wise t-statistics that compared theactivation for the affirmative condition andnegative condition to the baseline activationduring the sentence presentation using athreshold of t > 4.5.

Figure 18 shows two immediately successivebrain states for one participant, the first acquiredover approximately 3.5 s of sentence processing,the second over about 4 s of picture inspection,rotation, and comparison. Initially, duringsentence reading, there is significant activationin the left temporal area. There is also someactivation in the parietal regions, although theonset is slower. In both regions, the activationduring the sentence processing is greater for thenegative sentences than for the affirmativesentences. Then during picture inspection, thereis considerably more activation in the parietalregions, most of which was in and around theintraparietal sulcus (as illustrated in the Figure).These two images constitute a two-frame brainimage “movie” of sequential thought processes,showing the brain activation primarily first inone region and then primarily in another, as thenature of the cognitive activity changes.

Figure 18. Two immediately successive brainstates for one participant, the first acquired overapproximately 3.5 s of sentence processing, thesecond over about 4 s of picture inspection,rotation, and comparison. Initially, duringsentence reading phase, there is significantactivation in the left temporal area and someactivation in the parietal regions. During thesubsequent picture inspection phase, there isconsiderably more activation in the parietalregions, much of which is in and around theintraparietal sulcus (From “Movies of the brain:

Imaging a sequence of cognitive processes” byJust, Carpenter, Keller, & Thulborn, 1996,NeuroImage, 3, S250. Copyright 1996 byAcademic Press. Reprinted with permission).

This asymptotic picture of how the activationshifts does not convey an important feature ofthe activation dynamics. Namely, even duringthe sentence comprehension phase, theactivation in both the left posterior temporalregion and the parietal regions was significantlyabove the baseline. Figure 19 shows the timecourse of the activation in the left posteriortemporal region (the left panel) and the leftparietal region (the right panel) during thesentence comprehension phase. Note that evenwhen the participants were just beginning toread the sentence, there was significantactivation in the parietal region. Moreover, inboth regions, the activation was greater fornegative than affirmative conditions, indicatingthat the impact of linguistic difficulty extends toregions that process the spatial information towhich the sentences referred. Thus, thedynamics are more subtle than simply an on-offswitch associated with each major processingstage. Rather, the co-activation of these regionssuggests a more continuous cascade ofactivation among communicating corticalregions. The co-activation suggests that thesecomponent systems, such as the linguisticcomprehension system and the visuo-spatialsystem, are only somewhat independent and,depending on the task, can also work together ina closely coupled way.


20

Figure 19. The time course of activation in the left posterior temporal region (the leftpanel) and the left parietal region (the right panel) during sentence reading for each ofseveral acquisition images. Both regions show significant activation above baseline, andthe activation is greater for negative than affirmative sentences, suggesting that sentenceprocessing affects the visuo-spatial network in the parietal region while the sentence isstill being comprehended.

The Capacity Utilization Model

Cognitive computations are neurally instantiatedthrough a complex biochemical process that canbe seen as a form of energy transformation. Likeany biological energy process, thistransformation requires resources. Moreover,the demand for and consumption of resources atthe neural level can be abstractly mapped ontoresource consumption in computational modelsof cognitive processes. From this perspective,the fundamental resource assumption of thecurrent theory is that immediate thought issupported by a set of limited, system-specificresources that enable the maintenance ofrepresentations and the cognitive operationsthemselves. Fortunately for functionalneuroimaging, brain activation reflects at leastone aspect of the resource consumption

engendered by the cognitive activity. At amacro-level, the entire cognitive system iscomposed of several major systems, each ofwhich is supported by its own resource pool.For example, the cortical neural systemscorresponding to the language and spatialsystems consist of several anatomically distinctand somewhat separated subsystems. Forexample, language is partially subserved bycortical regions in the frontal and in the temporallobes, more actively so in the left hemisphere,but also in the right. Each subsystem may befurther decomposable, at several temporal grains(in the computational model) and severaltemporal and spatial grains (in various neuro-imaging data sets).

Within a system, such as the languageprocessing system, demand on the resources


21

arises in part from keeping the activation levelof a representational element above some pre-specified threshold and processing. Thecomputations that are performed by thecognitive system are dependent on the sameresource pool that is drawn on to maintain theactivation of representational elements. Thecomputations are accomplished by activation-manipulating production rules (if-then rules orcondition-action contingencies). If the conditionelements of a rule have an adequately highactivation level, the rule performs its function bypropagating activation to its action elements. Aparsing production rule, for example, might haveas its condition the encoding of a definite article(the) and as its action, increasing the activationassociated with the representation of a nounphrase.

The supply of the activation resources in thesystem is limited. If the productions require alarge amount of activation to complete theirfunctions, for example, if there are manyelements to keep active and difficultcomputations to complete, then the demands onthe resource pool may begin to outstrip thesupply. In that case, when a production fires itwill propagate less activation than it wouldotherwise. Its propagation of activation is scaledback in proportion to the amount of activationshortage. This slows the processing rate byrequiring the production to fire over more cyclesto activate its target to a given level. Activationalso may be conserved by de-allocating some ofthe activation associated with previouslyactivated elements, producing gradualforgetting.

Furthermore, the size of the supply of resourcesfor a system is assumed to vary among normalindividuals and to be a source of individualdifferences in cognitive performance.Concretely, the neural resources is a broadcategory that includes neurotransmitters’functioning, the various metabolic systemssupporting the neural system, and also thestructural connectivity or integrity of the neuralsystems, drawing directly from the concept ofneural-systems efficiency (Parks et al., 1989).In the functional analysis, these can be abstractlymapped onto the aggregate concept of functionalresources that enable various cognitivecomputations. While the theory provides aframework for considering individual

differences, it does not account for the origin ofsuch differences, which presumably arise froman interaction of biological and environmentalfactors (such as experience and training). Inaddition, pathological neurological changes inan adult, such as those associated with strokes(Thulborn, Carpenter & Just, in press) ordementia (Thulborn, Martin, Sweeney, 1998),may be viewed as reducing the resource supply.

Defining capacity utilization.

Within this framework, the amount of demandthat a task imposes on a resource pool can bedefined as the number of units of activationrequired over a given time interval to performthe task. This quantity can be measured in thecourse of the model’s performance of the task.This framework provides a concreteoperationalization of capacity utilization, aconcept that was originally proposed ineconomics and refers to the proportion ofresources that a system uses in a given timeinterval. For example, if a manufacturing plantoperates for 8 hours one day but is capable ofoperating a maximum of 24 hours per day, thenthe capacity utilization is 33% for that day.Analogously, if an individual needs 15 units ofactivation to perform a cognitive task and has atotal of 60 units of activation available, thecapacity utilization for the task is 25%.Capacity utilization, thus, is conjointlydetermined both by the amount of resourcesrequired by the task (i.e., demand) and theamount of resources available (i.e., supply). Fora given person, the more demanding the task is,the more activation should be observed,provided the task difficulty remainsperformable. This outcome was observed in thesentence comprehension and mental rotationstudies described elsewhere in this article. For agiven level of task difficulty (i.e. keeping thedemand constant), the smaller the resourcesupply is across a range of individuals (due tolower skill level, brain damage, or somebiological factor), the greater should be theamount of activation. This result has beenobtained in several studies, one of whichobserved a high negative correlation between theamount of brain activation during theperformance of visual analogies (from anintelligence test called the Raven ProgressiveMatrices Test) and the test score (Haier, 1993).


22

That is, the lower the test score, the greater theamount of observed activation.

The construct of capacity utilization focuses on adimension that has been difficult to measure andthat is often neglected in cognitive theory inspite of its importance: a characterization of howhard a cognitive system has to work to perform agiven task (Just, Carpenter, Keller, Eddy &Thulborn, 1996). It seems introspectivelyobvious that our minds work much harder atsome times than at other times. This notion ofthe mind’s “working harder” is concretized inthe model, which provides mechanisms formapping from processing characteristics tocapacity utilization as well as to response timesand error rates.

Computational mechanisms.

Besides these assumptions about the resources,the theory also proposes a particular cognitivearchitecture that we will briefly describe. Thearchitecture is a hybrid with both symbolic andconnectionist features. The connectionistmechanisms have to do with the activation-based processing and storage, that gives eachfunction a graded quality. The symbolicmechanisms consist of condition-action rulesthat express procedural knowledge. As theactivation of elements in a condition side of arule reaches a threshold, the productionincrements the activation of the action elements.The processing is graded in the sense thatproduction rules fire reiteratively, usually untilthey attain their goal. Thus, conditions andactions are linked by productions that propagateactivation to action elements. The productionsare like dynamically formed links betweennodes in a connectionist network (Just,Carpenter, & Varma, 1999). The condition-action rules permit an organization of high levelprocessing (relating high level goals and theactions required to attain them, as in problemsolving) as well as lower-level associativeprocessing. The production rules self-schedulethemselves based on the emergence of theirenabling conditions, without any central controlmechanism (such as a central executive).Furthermore, the processing can be parallel, inthat all satisfied productions can fire in parallelin a given cycle if resources permit it, orprocessing can be serial, if the initiation of some

production is dependent on the activation of aproduct of some preceding computation.

The theory focuses on capacity utilization at aparticular system level that aggregates overlower-levels processes that undoubtedly existand may even set parameters on the efficiency ofprocesses at the higher-level. For example, theactivation to threshold of a particular set ofrepresentations likely entails the inhibition ofcompeting representations, although that is notyet incorporated into the modeling or in thismapping. Nevertheless, the architecture reflectsa beginning theoretical effort to focus on thedynamics of resource modulation duringimmediate thought.

A new version of the modeling system,tentatively called 4CAPS, is currently underdevelopment. Its main innovation is that 4CAPScontains a number of embedded productionsystems which correspond to large-scale corticalneural networks. Each embedded productionsystem has its own time scale and its own set ofordered preferences of processing style, but theprinciples of operation are common across thesubsystems. The goal is to simulate not only thecontent, time course, and errorfulness of humanperformance in various high level cognitivetasks, but also the relative amount of neuralactivation that the task engenders in each ofseveral neural components/subsystems ofcognition.

Clinical applications take capacity utilizationinto account.

The pragmatic implication is that in the fMRItesting of any clinical case, it is important tomeet a set of conditions. First, the patient mustbe effectively performing the cognitive task, asassessed by concurrent behavioral measures,such as response accuracy. There is usually arange of task difficulty levels that a given patientcan effectively perform, defining a dynamicrange of that person’s system for that type oftask. If the task is too difficult to perform (suchthat the response accuracy approaches chancelevel), then the amount of brain activationdecreases relative to easier, performable levelsof the task. Such decreases are not paradoxicalnon-monotonicities in the effect of cognitiveworkload, but simply reflect a kind of “givingup” on the task, and hence display a decrease in


23

brain activation. A second important condition isthat the task should probe several points alongthe dynamic range of the tested system,extracting the counterpart of a dose-responsecurve. Tasks that are so easy that they do notadequately engage the system may fail to revealhigher-level dysfunctions. For example, to test“language” ability in patients, single-wordidentification or word-reading tasks aresometimes used. In the case of high-functioningindividuals with autism, there is no deficitwhatsoever compared to control subjects in theability to process single words (Minshew,Goldstein & Siegel, 1997). In fact, the high-functioning individuals with autism generallyperform better than the matched controls onthese tasks. However, when the same two groupsare compared in a complex sentencecomprehension task, then deficits can be foundin the autistic patients. Any fMRI comparison ofother patient groups should similarly examinebrain function at several levels of difficulty, toensure that there is no resource deficit that mightbe observable only at difficult (yet stillperformable) levels of the task.

Localization of function.

Many researchers consider functionalneuroimaging to be a localization tool, toassociate psychological function with locationsof brain activation. However, the issue oflocalization is more complex and moreinteresting than this first-order researchsuggests. First, for every psychological functionthat is executed, multiple brain areas activate.Second, the mapping should be frompsychological function to a pattern of brainactivation, not to a set of locations. To be sure,the pattern must apply to a set of locations, butultimately, a psychological function will notmap to a single location or to a single neuron.Finally, brain-imaging findings indicate that themapping between psychological function andbrain activation is dynamic, such that the size ofthe location and number of locations activatedchanges with the amount of computationaldemand. The computational demandsthemselves are dynamic, changing over a shorttime period within the task and over a longertime period, with factors such as learning,fatigue, and so forth.

The classical view of localization attempts toassociate each cortical region with a singlecognitive process. Such a view leads to thephrenological models of the brain, updated toinclude more cognitive functions in place ofpersonality attributes, but essentially proposing aone-to-one map of structure to function. Arelatively recent version of this approach todescribing cortical function is depicted in Figure20. Implicitly, this view often assumes that theprocessing from one stage to the next is serialand sequential, without room for feedback. Forexample, the Wernicke-Geschwind model ofword reading and pronunciation, shown inFigure 20, proposed that written words are firstdecoded in the visual areas, then therepresentation was transmitted to the posteriortemporal region for interpretation, and then ontothe frontal region for translation into a spokencode that was translated into the motor output.With this view, the activation of multipleregions simply reflects the view that cognitivetasks are complex and involve multiple basiccognitive processes.

Figure 20. A depiction of a one-to-one mappingthat is conventionally postulated betweencognitive processes and brain areas. Eachprocess is associated with a single brain region,and regions are assumed to serially processinformation from input to output, as indicated bythe curved black arrows. For example, theGeschwind-Wernicke model proposed that inorder to pronounce a printed word, it was firstencoded in the visual areas, then interpreted inWernicke’s area, and then translated into apronounceable code in Broca’s area. The


24

alternative view, schematically depicted inFigure 1 and supported by the very high fieldstudies described in this article, suggests thatthere is a many-to-many mapping betweencognitive computations and cortical regions.

The alternative perspective that has beenexamined in the reported research is that themapping between function and cortical area isnot one-to-one, but one-to-many. That is, acognitive computation is subserved by morethan one region, and moreover, a region mayparticipate in multiple computations, althoughwith different weights. Moreover, theconnections among regions are not simply oneway; the studies reported here and elsewheresuggest that cognition is supported by systemsthat are linked by extensive feedback as well asfeedforward connections. One immediateimplication is that brain damage impairs thefunction of not just the lesioned area, but of allthe networks which include the damaged area.As a result, the emerging map between cognitivefunction and neural region is much morecomplex, highly interactive and non-localized,as the sketch in Figure 1 begins to depict.

Functional connectivity.

Referring to a large-scale network implies thatthere is a considerable degree of interactionamong the network components. The network isidentified in the first place by the co-activationof its members. Neuroimaging provides severaltypes of further evidence of interaction, beyondco-activation. First, brain activation can revealco-modulation of two or more networkcomponents by the same independent variable.For example, variation in the syntacticcomplexity of sentences similarly affectsBroca’s and Wernicke’s areas, suggesting that

they might be interacting. Second, the precisetime course of the activation can be compared,to reveal similarities in activation time courseacross disparate areas of the brain. If there is acluster of voxels defined by having similar timecourses, and that cluster is distributed over twoor more components of the network, then it islikely that the participating voxels in thosecomponents are interacting. New statisticalmethods and software tools (such as Evident) arebeing developed to cluster voxels based on thesimilarity of the time course of their activation.The methods identify network members on thebasis of not just coactivation, but on coactivationwith the same time course. This generalapproach for identifying network coordinationhas been called functional connectivity (Horwitz,1998).

We have begun such functional connectivityanalysis in some sentence comprehensionstudies. The data are analyzed by finding thevoxels whose activations are time-locked to eachother and grouping all the voxels with a similartime course irrespective of their location. It ispossible to identify a single cluster of voxels thatcorresponds to the entire set of activated voxelsin the t-map or statistical probability map of thetype shown throughout this chapter. However, ifwe further refine the clustering, it is possible tofurther subdivide the entire group of activatedvoxels into subgroups that constitute functionalsub-networks. The identified sub-networks arevery meaningful. The result for one participantin a 3.0T language study, in Figure 20, showsthe early promise of this approach, identifying asub-network that includes a subset of the voxelsin Wernicke’s area to those in the left prefrontalcortex.


25

Figure 21. A sub-network extracted for one participant in a 3.0T language study, showsa subset of voxels in Wernicke’s area and the left prefrontal cortex (indicated by circles).These voxels share a similar time course of activation throughout the languagecomprehension paradigm, suggesting that they are functionally connected.

Information from brain imaging aboutcognition.

All imaging reveals some spatial pattern, andfMRI is no different at a general level. However,the interpretation of fMRI provides considerableadditional information. fMRI reveals thelocation of activated parts of the brain, and moreimportantly, it reveals the co-activation ofcortical components in a network. With the helpof time-series-based clustering techniques, it canalso reveal functional connectivity in thenetwork. Additionally, fMRI reveals the amountof activation in each network component in aparticular task condition, reflecting conjointlythe distribution of computational demands

imposed by the task and the distribution ofneural (biological and cognitive) resourcesbrought to bear by the subject’s brain. fMRI canalso reveal the time course of brain activation,within the temporal limits of resolution imposedby the T1 relaxation times (on the order of 1sec). fMRI also reveals how a damaged networkresponds after adaptation, providing ampleclinical application of the technique. But thefundamental conceptual insights fostered byfMRI technology in combination withdeveloping theories of cognitive functionilluminate the dynamic and interactive brainnetworks that subserve thinking and thatconstitute the mind.


26

Acknowledgement:

This work was supported in part by NationalInstitute for Neurological Disorders and StrokeGrant P01 NS35949-01A1, National Institute ofMental Health Grant MH-29617, Office ofNaval Research Contract N00014-96-1-0322,and National Institute of Mental Health SeniorScientist Awards MH-00661 and MH-00662.We would like to thank Tim Keller and KeithThulborn who have been excellent collaboratorson many of these studies.

References

Alivisatos B, Petrides M. Functional activationof the human brain during mental rotation.Neuropsychologia. 1997;35:111-118.

Arnoult MD. Shape discrimination as a functionof the angular orientation of the stimuli.Journal of Experimental Psychology.1954;47:323-328.

Bottini G, Corcoran R, Sterzi R, et al. The roleof the right hemisphere in the interpretationof figurative aspects of language: Apositron emission tomography activationstudy. Brain. 1994;117:1241-1253.

Buckner RL, Koutstaal W, Schacter DL, WagnerAD, Rossen BR. Functional-anatomic studyof episodic retrieval using fMRI.NeuroImage. 1998;7:151-162.

Carpenter PA, Just MA, Keller TA. The corticalnetworks supporting reading and listeningcomprehension. Unpublished manuscript,Department of Psychology, CarnegieMellon University, Pittsburgh, PA. 1999.

Carpenter PA, Just MA. Sentencecomprehension: A psycholinguisticprocessing model of verification.Psychological Review. 1975;82:45-73.

Carpenter PA, Just MA, Keller TA, Eddy. W. F.,Thulborn KR. Graded functional activationin the visuo-spatial system with the amountof task demand. Journal of CognitiveNeuroscience. 1999a;11:9-24.

Carpenter PA, Just MA, Keller TA, Eddy. W. F.,Thulborn KR. The time course of fMRI-measured activation in language and visuo-spatial networks, 1999b, under review.

Carroll JB. Human Cognitive Abilities: A Surveyof Factor-Analysis Studies. New York:Cambridge University Press.; 1993.

Chase WG, Clark HH. Mental operations in thecomparison of sentences and pictures. In:

Gregg L, Ed. Cogntion in Learning andMemory. New York: Wiley; 1972.

Cohen MS, Kosslyn SM, Breiter HC, et al.Changes in cortical activity during mentalrotation: A mapping study using functionalMRI. Brain. 1996;119:89-100.

Evident 4.22 [Computer Software] Winnipeg,Manitoba, Canada: National ResearchCouncil of Canada, Institute forBiodiagnostics.

Farah MJ. Visual Agnosia: Disorders of ObjectRecognition and What They Tell Us AboutNormal Vision. Cambridge, MA: MITPress; 1990.

Haarmann HJ, Just MA, Carpenter PA. Aphasicsentence comprehension as a resourcedeficit: A computational approach. Brainand Language. 1997;59:76-120.

Haier RJ. Cerebral glucose metabolism andintelligence. In: P. A. Vernon, Ed.Biological Approaches to the Study ofHuman Intelligence. Norwood, NJ: Ablex;1993:317-332.

Horwitz B. Using functional brain imaging tounderstand human cognition. Complexity.1998;3:39-52.

Jurden FH. Individual differences in workingmemory and complex cognition. Journal ofEducational Psychology. 1995;87:93-102.

Just MA, Carpenter PA. Eye fixations andcognitive processes. Cognitive Psychology.1976;8:441-480.

Just MA, Carpenter PA. The intensity ofthought: Pupillometric indices of sentenceprocessing. Canadian Journal ofExperimental Psychology. 1993;47:310-339.

Just MA, Carpenter PA, Keller TA, Eddy WF,Thulborn KR. Brain activation modulatedby sentence comprehension. Science.1996;274:114-116.

Just MA, Carpenter PA, Varma, S. Computermodeling of high-level cognition and brainfunction. Unpublished manuscript.Department of Psychology, CarnegieMellon University, Pittsburgh, PA. 1999.

Karbe H, Herholz K, Szelies B, Pawlik G,Wienhard K, Heiss WD. Regionalmetabolic correlates of Token test results incortical and subcortical left hemisphericinfraction. Neurology. 1989;39:1083-1088.

Kempler D, Curtiss S, Metter EJ, Jackson CA,Hanson WR. Grammatical comprehension,aphasic syndromes, and neuroimaging.


27

Journal of Neurolinguistics. 1991;6:301-318.

King J, Just MA. Individual differences insyntactic processing: The role of workingmemory. Journal of Memory andLanguage. 1991;30:580-602.

Lynch JC, Mountcastle VB, Talbor WH, YinTCT. Parietal lobe mechanisms for directedvisual attention. Journal ofNeurophysiology. 1977;40:362-389.

Mesulam M-M. Large-scale neurocognitivenetworks and distributed processing forattention, language and memory. Annals ofNeurology. 1990;28:597-613.

Metter EJ, Hanson WR, Jackson CA, et al.Temporoporietal cortex in aphasia:Evidence from positron emissiontomography. Archives of Neurology.1990;47:1235-1238.

Minshew NJ, Goldstein G, Siegel DJ.Neuropsychologic functioning in autism:Profile of a complex informationprocessing disorder. Journal of theInternational Neuropsychological Society.1997;3:303-316.

Parks RW, Crockett DJ, Tuokko H, et al.Neuropsychological "systems efficiency"and positron emission tomography. Journalof Neuropsychiatry. 1989;1:269-282.

Rademacher J, Galaburda AM, Kennedy DN,Filipek PA, Caviness VSJr. Human cerebralcortex: Localization, parcellation, andmorphometry with magnetic resonanceimaging. Journal of CognitiveNeuroscience. 1992;4:352-374.

Shah P, Miyake A. The separability of workingmemory resources for spatial thinking andlanguage processing: An individualdifferences approach. Journal ofExperimental Psychology: General.1996;125:4-27.

Shepard RN, Metzler J. Mental rotation of three-dimensional objects. Science.1971;171:701-703.

Tagaris GA, Kim S-G, Strupp JP, Andersen P,Ugurbil K, Georgopoulos AP. Mentalrotation studied by functional magneticresonance imaging at high field (4 Tesla):Performance and cortical activation.Journal of Cognitive Neuroscience.1997;9:419-432.

Thulborn KR, Carpenter PA, Just MA. Plasticityof language-related brain function duringrecovery from stroke. Stroke. in press.

Thulborn KR, Chang SY, Shen GX, VoyvodicJT. High resolution echo-planar fMRI ofhuman visual cortex at 3.0 Tesla. NRM inBiomed. 1997;10:183-190.

Thulborn KR, Martin C, Sweeney JA.Functional MRI with visually-guidedsaccades separates probable Alzheimer'sdisease from other dementias. In:Proceedings of the 28th Annual Meeting ofSociety for Neuroscience: 1998; LosAngeles, CA. 1998:Abstract 503.6 (pp.1269).

Thulborn KR, Voyvodic J, McCurtain B, et al.High field functional MRI in humans:Applications to cognitive function. In: P.Pavolone, P. Rossi, Eds. Proceedings of theEuropean Seminars in Diagnostic andInterventional Imaging, Syllabus 4. NewYork: Springer-Verlag; 1996:91-96.

Thulborn KR, Waterton JC, Matthews PM,Radda GK. Oxygenation dependence of thetransverse relaxation time of water protonsin whole blood at high field. Biochimica EtBiophysica ACTA. 1982;714:265-270.

Tompkins CA, Mateer CA. Right hemisphereappreciation of prosodic and linguisticindications of implicit attitude. Brain andLanguage. 1985;24:185-203.

Ungerleider LG, Haxby JV. 'What' and 'where'in the human brain. Current Opinion inNeurobiology. 1994;4:157-165.

Ungerleider LG, Mishkin M. Two cortical visualsystems. In: D. J. Ingle, M. A. Goodale, R.J. W. Mansfield, Eds. Analysis of VisualBehavior. Cambridge, MA: MIT Press;1982:549-586.

Modeling the Mind: High-Field fMRI-Activation …...Topics in Magnetic Resonance. Imaging (1999),...

Documents

Transcript of Modeling the Mind: High-Field fMRI-Activation …...Topics in Magnetic Resonance. Imaging (1999),...