Role of mental imagery in a property verification task

ROLE OF MENTAL IMAGERY IN A PROPERTY

VERIFICATION TASK: FMRI EVIDENCE FOR

PERCEPTUAL REPRESENTATIONS OF CONCEPTUAL

KNOWLEDGE

Irene P. KanUniversity of Pennsylvania, Philadelphia, USA

Lawrence W. BarsalouEmory University, Atlanta, GA, USA

Karen Olseth SolomonWillamette University, Salem, MA, USA

Jeris K. Minor and Sharon L. Thompson-SchillUniversity of Pennsylvania, Philadelphia, USA

Is our knowledge about the appearance of objects more closely related to verbal thought or to percep-tion? In a behavioural study using a property verification task, Kosslyn (1976) reported that there areboth amodal and perceptual representations of concepts, but that amodal representations may be moreeasily accessed. However, Solomon (1997) argued that due to the nature of Kosslyn’s stimuli, subjectsmay be able to bypass semantics entirely and perform this task using differences in the strength of associ-ation between words in true trials (e.g., cat–whiskers) and those in false trials (e.g., mouse–stinger).Solomon found no evidence for amodal representations when the task materials were altered to includeassociated false trials (e.g., cat–litter), which require semantic processing, as opposed to associative strat-egies. In the current study, we used fMRI to examine the response of regions of visual association cortexwhile subjects performed a property verification task with either associated or unassociated false trials.We found reliable activity across subjects within the left fusiform gyrus when true trials were intermixedwith associated false trials but not when true trials were intermixed with unassociated false trials. Our datasupport the idea that conceptual knowledge is organised visually and that it is grounded in the percep-tual system.

One of the leading theories of the organisation ofsemantic knowledge holds that conceptual infor-mation is distributed across distinct attributedomains, such as vision, touch, and action (Allport,

1985). Numerous investigators have drawn distinc-tions between visual knowledge and nonvisual (e.g.,functional) knowledge, in areas ranging fromlanguage acquisition (e.g., Gentner, 1978; Nelson,

COGNITIVE NEUROPSYCHOLOGY, 2003, 20 (3/4/5/6), 525–540

2003 Psychology Press Ltdhttp://www.tandf.co.uk/journals/pp/02643294.html DOI:10.1080/02643290244000257

525

Requests for reprints should be addressed to Irene P. Kan, Department of Psychology, University of Pennsylvania, 3815 WalnutStreet, Philadelphia, PA 19104-6196, USA (Email: [email protected]).

Supported by grants from National Institute of Health (NIH R01 MH60414) and Searle Scholars Program awarded to STS, andgrant from National Science Foundation (SBR-9905024) awarded to LWB. We thank Jessica Pickard for assistance with data collec-tion, Geoff Aguirre and Joseph Kable for assistance with data analysis, and two anonymous reviewers for helpful comments on anearlier version of this manuscript.

Q1520–CN SI 6 / May 28, 03 (Wed)/ [15 pages, 2 tables, 4 figures, 0 footnotes]–S endings, c.f. [no comma].

1974) to object categorisation (e.g., Rosch, Mervis,Gray, Johnson, & Boyes-Braem, 1976). Manyneuroimaging studies have supported the distinc-tion between visual and nonvisual knowledge (seeThompson-Schill, 2003, for a review). One inter-pretation of these studies is that our knowledgeabout the visual attributes of an object is repre-sented differently from our knowledge of other,nonvisual attributes. This claim is related to afundamental question that has been debated byphilosophers (e.g., Locke, 1690/1959) andpsychologists (e.g., Kosslyn, 1994; Pylyshyn, 1981)for centuries: To what extent does conceptualknowledge rely on perceptual representations, asopposed to propositional, amodal representation?That is, are the underlying representations descrip-tive (i.e., more akin to verbal thoughts andnonperceptual) or are they depictive (i.e., percep-tual and possess visual-spatial qualities)? Until themid-twentieth century, the view that conceptualknowledge is perceptually based was widelyaccepted (e.g., Locke, 1690/1959; Price, 1953).However, this assumption was called into question,and the nature of the internal representations ofconceptual knowledge has undergone much scru-tiny over the past few decades.

One approach that researchers have taken toaddress these questions is to examine the processesinvolved in mental imagery. It is assumed that inorder to create a mental image of an object, access toconceptual knowledge is necessary. That is, in orderto create a mental image of an apple, one mustretrieve the stored knowledge about the concept“apple.” Thus, it seems that understanding thenature of mental imagery may help us address morefundamental questions about the representation oflong-term conceptual knowledge, especiallyknowledge about visual attributes. According todepictive theories of conceptual knowledge, mentalimagery involves the same representations andmechanisms normally used during visual percep-tion (e.g., Kosslyn, 1994; Shepard & Cooper, 1982;Zatorre, Halpern, Perry, Meyer, & Evans, 1996).That is, conceptual representations are grounded inthe perceptual systems and that information isrepresented in a spatial and pictorial format. Oneline of evidence in support of this theory came from

visual scanning experiments. In Kosslyn, Ball, andReiser’s (1978) classic study, subjects wereinstructed to focus their attention on one part of amental image and “move” that part to the locationof some other part of the image. They reported thatthe time taken to complete a mental scan wasdirectly proportional to the physical distancebetween the two locations. Kosslyn and colleaguesinferred from these findings that mental imageryand visual perception must share similarmechanisms.

However, the validity of these conclusions hasbeen questioned. For example, Pylyshyn (1981)argued that subjects interpreted the task demandsas simulating the use of visual-spatial knowledge.Similarly, Intons-Peterson (1983) suggested thatsubjects were responding to experimenters’ expec-tations, and that their performance was biased bythe experimenters’ presumptions. These investiga-tors have proposed that subjects’ expectationsresulted in performance that appeared to support aperceptual-based theory, even though the under-lying representations were not perceptual. Instead,Pylyshyn and others have argued that there is acomplete independence between the perceptualsystem and the conceptual system, such that whenan object is perceived, its visual features will berepresented in the perceptual system, and theconceptual system will reinterpret that informationas a list of propositional features. For example, onperceiving an apple, the perceptual system repre-sents it in a visual form, and the conceptual systemreinterprets that information into an amodalfeature list, with entries such as “shiny,” “red,”“round.” Proponents of propositional theoriesargued that since much of cognition depends on a“language of thought,” it is only natural that mentalimages are represented propositionally, and thatthis amodal list is retrieved when conceptualknowledge is accessed (Fodor & Pylyshyn, 1988;Pylyshyn, 1984). However, as Barsalou (1999)noted, nearly all evidence for amodal theories isindirect, relying either on theoretical arguments, oron the ability to implement amodal representationsformally and computationally.

The only exception seems to be Kosslyn’s (1975,1976) property verification experiments, in which

KAN ET AL.

526 COGNITIVE NEUROPSYCHOLOGY, 2003, 20 (3/4/5/6)

he investigated whether visual imagery and otherforms of internal representations can be differenti-ated. In this task, subjects were asked to determinewhether certain properties are true or false ofcertain concepts. For example, when presentedwith the concept word “cat,” subjects were asked toverify whether the property “claws” is true of thatconcept. To determine whether individuals wouldconsult amodal knowledge or perceptual knowl-edge when verifying properties, Kosslyn (1976)manipulated task instructions within the samegroup of subjects. For the first block of trials,subjects did not receive explicit instructions on howto perform the task. They were simply told to verifythe properties as quickly as possible without sacri-ficing accuracy. For the second block of trials,subjects were given explicit imagery instructions.They were told to first form a vivid mental imageand then to verify each property by consulting theirmental picture. Furthermore, Kosslyn systemati-cally varied two other factors: size of a property rela-tive to its concept and the associative strengthbetween a property and its concept.

Under neutral instructions, subjects were signif-icantly faster at verifying properties than under theimagery condition. Furthermore, reaction timescollected under the neutral instructions were corre-lated with associative strength between propertiesand their concepts, which support theories ofamodal representations. On the other hand, reac-tion times collected under the imagery conditionwere not correlated with associative strength;instead, the reaction times were inversely correlatedwith property size. These results support theories ofperceptual representations. In sum, Kosslynconcluded that there are both amodal and percep-tual representations of concepts, but that amodalrepresentations may be more easily accessed.

Although these findings have been cited exten-sively, recent work challenges the extent to whichthese findings can elucidate the representation ofconceptual information. Solomon (1997) ques-tioned the nature of the stimuli used in Kosslyn’s(1976) original studies (see also Barsalou, Solomon,& Wu, 1999), noting that the true trials and thefalse trials actually differed on two dimensions:(1) whether the properties belong to the concepts,

and (2) whether the properties are associated withthe concepts. One way of assessing associativestrength between two concepts is how frequentlythe two concepts co-occur in the same context. Themore frequently they occur together, the more asso-ciated they are. Another way of thinking aboutassociative strength is how readily a conceptbecomes available when another concept isaccessed. For example, consider the two concept–property pairs “cat–head” and “mouse–stinger.”The words in the first pair are associated becausethese words often appear together within the samecontext. On the other hand, the words “mouse” and“stinger” rarely occur within the same context, andit is unlikely that these two words are associatedwith each other. In other words, not only do theseword pairs differ on whether the property belongswith the concept, they also differ on associativestrength.

The inadvertent variation in associationstrength between false and true trials may have hadconsequences for the results reported by Kosslyn(1976). Solomon (1997) pointed out that while allof the true properties in Kosslyn’s original studywere associated to their concepts (e.g., cat–head),all of the false properties were unassociated to theirconcepts (e.g., mouse–stinger). It was possible forsubjects in the neutral condition to perform the taskbased on a simple word association strategy. Thatis, subjects might have adopted a strategy ofanswering “yes” if the concept and property are at allassociated and answering “no” if the concept andproperty are unassociated. Furthermore, it has beenshown that when using a simple word associationstrategy, access to conceptual knowledge is notobligatory (Glaser, 1992). Glaser proposed a modelthat includes a direct connection between theprinted words and the verbal lexicon, and this routeis independent from the conceptual system. Ifsubjects in Kosslyn’s experiment were in fact notaccessing conceptual knowledge to perform thetask in the neutral condition, then the findingsfrom that experiment do not bear on the questionof the nature of conceptual knowledgerepresentations.

Solomon (1997) replicated Kosslyn’s propertyverification experiments (1975, 1976), with one

COGNITIVE NEUROPSYCHOLOGY, 2003, 20 (3/4/5/6) 527

CONCEPTUAL KNOWLEDGE IN VISUAL CORTEX

important variation: To ensure that conceptualprocessing would be engaged in all property verifi-cation trials, Solomon replaced the unassociatedfalse trials with associated false trials (e.g., steak–grill) in one condition of the experiment. Solomonreasoned that subjects would not be able to rely on asimple word-association strategy when theconcepts and properties of both true and false trialswere equivalently associated. Under these condi-tions, subjects’ performance was predicted byperceptual factors (e.g., size) and not linguisticfactors (e.g., associative strength). Contrary toKosslyn’s (1976) results, Solomon obtained thispattern of data even when subjects received neutralinstructions (i.e., associated-neutral subjects). Theauthor interpreted these data as support for the ideathat conceptual knowledge is grounded in percep-tual simulation. The present experiment followsfrom this study, but additionally addresses differ-ences in neural activity between associated andunassociated conditions.

One approach to investigating the relationbetween conceptual knowledge and perception hasbeen to compare neural mechanisms of each. Thisapproach has been motivated, in part, by the unre-solved questions from behavioural experiments.Neuropsychological studies of patients withperceptual and imagery deficits as a result of braindamage have been used to bolster the hypothesisthat mental imagery and perception share commonprocesses (e.g., Bisiach & Luzzatti, 1978; Farah,Soso, & Dasheiff, 1992; Servos & Goodale, 1995).Recently, functional neuroimaging has been usedto investigate this question in normal subjects.Some studies have reported activation in visualcortex during mental imagery (e.g., Charlot,Tzourio, Zilbovicius, Mazoyer, & Denis, 1992;D’Esposito et al., 1997; Goldenberg, Podreka,Steiner, Franzen, & Deecke, 1991; Goldenberg,Podreka, Steiner, & Willmes, 1987; Roland &Friberg, 1985), in extrastriate and occasionallyprimary visual areas (see Kosslyn & Thompson,2000, for a full review of this topic). As thisapproach will be used in the present investigation,we briefly review some relevant findings here.

Using functional magnetic resonance imaging(fMRI), D’Esposito and colleagues (1997) used a

mental image generation task to examine the neuralbases of mental imagery. Subjects listened to alter-nating blocks of concrete words (e.g., apple, horse)and abstract words (e.g., treaty, guilt) while in thescanner. Subjects were instructed to generatemental images of the concrete words’ referents, andthey were told to listen passively to the abstractwords. D’Esposito and colleagues reported that inneurologically intact adults, visual associationcortex was recruited during mental imagery.Specifically, when compared to passive listening ofabstract words, generating images in response toconcrete words resulted in significantly greateractivity that was asymmetrically lateralised to theleft fusiform gyrus, left premotor area, and the leftanterior cingulate gyrus. These data were alsoconsistent with other neuroimaging studies ofmental imagery (e.g., Goldenberg et al., 1987,1991).

Thompson-Schill, Aguirre, D’Esposito, andFarah (1999) reported similar findings in theirinvestigation of the neural bases of semantic knowl-edge using fMRI. Subjects in their experiment wereasked a series of yes/no questions about visual ornonvisual characteristics of living and nonlivingobjects. For example, the following questions wereposed to the subjects: visual/living—“Does a parrothave a curved beak?”; nonvisual/living—“Arepandas found in China?”; visual/nonliving—“Arebows of violins longer than violins?”; and non-visual/nonliving—“Does a toaster use more elec-tricity than a radio?” Consistent with D’Espositoet al.’s (1997) results, Thompson-Schill andcolleagues also found differential patterns of activa-tions between visual knowledge retrieval andnonvisual knowledge retrieval. Specifically, greaterleft fusiform gyrus activity was found when subjectsengaged in visual knowledge retrieval than whenthey executed nonvisual knowledge retrieval. Addi-tionally, in one subject who participated in bothexperiments, when activation during visual knowl-edge retrieval was compared to the explicit imagerytask used by D’Esposito et al., considerable overlapin the left fusiform gyrus was found between thesetwo tasks. On the basis of these results, the leftfusiform gyrus was a crucial region of interest in thepresent investigation.


KAN ET AL.

The aim of the present study was to demonstratethe involvement of visual cortical regions duringsemantic processing—even with no explicitimagery instructions—and furthermore to showthat such involvement depends on the extent towhich semantic knowledge, as opposed to lexicalassociation information, is required to perform thetask. Specifically, we used fMRI to examine theinvolvement of the left fusiform gyrus in the prop-erty verification task. Activity in the visual associa-tion cortex during property verification wouldsupport the assertion that semantic knowledge isindeed grounded in the perceptual system.However, this prediction only applies to tasks thatrequire the retrieval of semantic knowledge. Thus,we hypothesised that different patterns of activa-tion would be observed depending on whether thefalse trials in the experiment were associated orunassociated pairs. To test this hypothesis, subjectswere randomly assigned to the associated condi-tion, in which both true and false trials were associ-ated pairs (as in Solomon, 1997), or theunassociated condition, in which true trials wereassociated but false trials were unassociated pairs (asin Kosslyn, 1976). Following from Solomon, wepredicted that left fusiform activity during propertyverification would only be observed in the associ-ated condition because this condition forcessubjects to retrieve semantic knowledge. Incontrast, activation of the left fusiform gyrus duringproperty verification would not be expected to reli-ably occur in the unassociated condition, in whichsubjects are able to adopt a word associationstrategy to perform the task. If semantic knowledgeis represented amodally (Pylyshyn, 1981, 1984),then no activation in the visual association cortex isexpected in either condition.

METHODS

Subjects

Fourteen subjects from the University of Pennsyl-vania participated in this experiment. Subjects wererandomly assigned to either the associated or unas-sociated task condition. Subjects in the associated

group were two males and five females, aged 19–21years (mean age = 19.9 years). Subjects in the unas-sociated group were five males and two females,aged 18–22 years (mean age = 20.6 years). Allsubjects met the following inclusion criteria:They were (1) high-school educated, (2) nativeEnglish speakers, and (3) right-handed. Generalexclusionary criteria were (1) history of neurologicalor psychiatric illness or (2) current use of medica-tion affecting the central nervous system (e.g.,psychotropic drugs). Informed consent wasobtained from all subjects. Each subject was paid$20 for his or her participation.

Materials

Property verification taskEach subject performed a property verification task,in which the task was to determine whether a prop-erty (e.g., frosting) was part of a concept (e.g., cake).Subjects were told that each stimulus pair wasarranged in a top–bottom configuration and that allconcept words were presented on the top and allproperty words were presented on the bottom. Allof the concept–property pairs used in this experi-ment were originally developed by Solomon(1997). To construct the 100 true trials, 100 prop-erties (e.g., hood, udder, drawer) were paired with100 concepts (e.g., car, cow, dresser) from 18different superordinate categories (e.g., vehicle,mammal, furniture), and all properties were limitedto physical parts of concepts. For example, “hill”could be loosely considered a property of “ant,” buthere it was considered a false property because itwas not a physical part of the concept. These pair-ings varied widely along both association strengthand size of property, and no property or concept wasever repeated.

For the false trials, two sets of materials wereconstructed. The associated set was constructed bypairing 100 concepts with 100 associated falseproperties (i.e., canary–sing). The unassociated setwas constructed by pairing the same 100 conceptswith a random order of the same 100 false proper-ties (i.e., canary–wine), thereby creating unassoci-ated pairs, but also keeping the concepts and



properties constant. No false property or conceptwas repeated or used as a true property or concept.

To assess the association strength of both thetrue and false pairs, an independent group of 20subjects from University of Chicago rated the asso-ciation strength of all possible pairs. For this task,subjects first read a concept word, then a propertyword, and then rated how much the property wordhad come to mind while reading the concept word.Subjects made their rating from a scale of 1 (did notcome to mind) to 9 (immediately came to mind). Thetrue pairs varied widely in association strength, withthe 50 most-associated properties averaging 7.5and the 50 least-associated properties averaging4.6. The two kinds of false pairs differed markedly,with the associated false pairs averaging 5.0 and theunassociated false pairs averaging 1.2.

True pairs were also chosen to represent a widevariety of sizes of properties of the concepts. Forexample, a cat’s head is larger than a cat’s claw andtherefore a head should be easier to resolve on animage of a cat than a claw. To assess the size of theproperty on the concept, 12 new subjects imaginedeach concept and estimated the percentage of theconcept that the property occupied. For example,subjects were asked, “What percentage of the totalvolume of a fox is its nose?” Subjects gave a ratingfrom 0 to 100 for each concept–property pair. Thelargest 50 properties averaged 29.7%, whereas thesmallest 50 properties averaged 10.6%. Concept–property pairs were also chosen so that associationstrength and the size of the property was not corre-lated (r = .13, p > .25).

To summarise, there were three types of prop-erty verification trials—true, associated false, andunassociated false. The correct response for a truetrial was “yes,” and the correct answer for a false trialwas “no.” Subjects in both the associated and theunassociated groups encountered the same truetrials (e.g., goat–ear). The critical trials that differedbetween the two groups were the false trials.Subjects in the associated condition received asso-ciated false trials (e.g., stapler–paper), and subjectsin the unassociated condition received unassociatedfalse trials (e.g., stapler–vegetable). Although bothassociated false trials and unassociated false trialsrequired a “no” response, the relationships between

the stimulus words differed in the two conditions.In the associated false trials, even though the twowords were associated with each other, the propertywas not part of the object, but in the unassociatedfalse trials, the property was neither part of theobject nor an associate of the concept.

Baseline letter verification taskIn the letter verification task, subjects determinedwhether a single letter (e.g., “m”) was part of apronounceable nonword letter string (e.g.,“smalum”). As in the property verification task, thestimulus pair was arranged in a top–bottom config-uration, where the pronounceable nonwordappeared on the top and the single letter appearedon the bottom. Average number of letters in thenonwords was matched to those of the experi-mental trials.

Procedures

The procedure used in the associated and unassoci-ated groups was identical, with the only differencebeing the stimuli in the critical false trials, asdescribed above. Each trial began with a fixationpoint presented in the middle of the screen for 200ms. The concept word (or nonword) followed thefixation and was presented for 2800 ms andremained on the screen for the duration of the trial.The stimulus onset asynchrony between theconcept word (or nonword) and the property word(or letter) was 1300 ms. In other words, the totalexposure time for the concept word (or nonword)was 2800 ms, and the total exposure time for theproperty word (or letter) was 1500 ms. The dura-tion of each trial was 3000 ms. Subjects indicatedtheir response with bilateral button presses on afour-button keypad, and their response times inmilliseconds were recorded. If the answer was “yes,”they pressed the two inner buttons, and if theanswer was “no,” they pressed the two outerbuttons. They were told to respond as quickly aspossible without sacrificing accuracy, and they weretold that they must respond within the 1500 mstime window during which the property wordremained on the screen. It is important to note thatsubjects were not given explicit instructions on how

KAN ET AL.


to make their decision (e.g., no imagery instruc-tions were given); rather they were told to deter-mine as accurately and as quickly as possiblewhether the property was part of the concept. Foursample items, using nontest pairs, were shown tothe subjects to familiarise them with the task priorto the scanning session.

The timing and trial composition of the blockswere the same for both tasks. Each experimentalrun consisted of 20 blocks of trials, alternatingbetween property verification blocks and letter veri-fication blocks. True and false trials were inter-mixed within each property verification block (seeFigure 1). There were 10 blocks (i.e., 100 trials) oftrials in each run. There were 50 property verifica-tion trials and 50 letter verification trials. Among allof the trials within each run, half of them required a“yes” response, and half of them required a “no”response. A total of four runs (i.e., 400 trials)comprised the experiment.

Using PsyScope software (Cohen,MacWhinney, Flatt, & Provost, 1993), stimuliwere presented by a Macintosh G3 Powerbookconnected to an LCD projector. The image wasprojected to a screen that was placed at the subjects’feet. A mirror that was mounted to the head coilwas placed right above the subjects’ eyes, and itallowed them to see the projected image on the

screen. However, a different stimulus presentationsystem was used for three of the subjects (one in theassociated group and two in the unassociatedgroup). For these three subjects, the Avotec SilentVision Visual Presentation system was used(Avotec, 2002). A four-button fibre-optic responsepad connected to the computer was used to recordthe subjects’ button press responses.

Image acquisitionFollowing the acquisition of saggital and axial T1-weighted localiser images, gradient echo,echoplanar fMRI was performed in 21 contiguous5 mm axial slices (TR = 2000, TE = 50, 64 × 64pixels in a 24 cm field of view, voxel size = 3.75 mm× 3.75 mm × 5 mm) using a 1.5-T GE Signa systemequipped with a fast gradient system and the stan-dard quadrature head coil. To minimise headmotion, foam padding was placed between thesubject’s head and the head coil. Twenty seconds of“dummy” gradient and rf pulse preceded the actualdata acquisition to approach steady-statemagnetisation.

Image processingOffline data processing was performed usingVoxBo software (VoxBo, 2002). After imagereconstruction, the data were sinc interpolated intime to correct for the fMRI acquisition sequence.A slicewise motion compensation method wasutilised to remove spatially coherent signal changesby the application of a partial correlation method toeach slice in time (Zarahn, Aguirre, & D’Esposito,1997). Additional motion detection and correctionwas undertaken using a six-parameter, rigid-bodytransformation. None of the subjects hadtranslational motion that exceeded 2 mm in anyplane or angular motion that resulted in more thana 2 mm displacement. Additionally, spatialsmoothing and normalisation were performed.Raw data for all runs from each subject were trans-formed to standardised MNI space (Evans, Collins,Mills, Brown, Kelly, & Peters, 1993) and spatiallysmoothed by convolution with a three-dimensionalGaussian kernel that has a FWHM of 1.5 × 1.5 ×2.0 (in voxels).



GOAT

ear

SAW

tool

STAPLER

paper

ROSE

thorn

JAMAFFU

b

LUMT

m

STAPLER

vegetable

ROSE

thorn

SAW

rodent

GOAT

ear

LUMT

m

JAMAFFU

b

...

...

...

...

ASSOCIATED UNASSOCIATED

BaselineLetter trials

Property

Verification trials

Figure 1. Pictorial depiction of trial types.

Image analysesVoxelwise analysis was performed on eachsubject’s data by using a general linear model forserially correlated error terms (Worsley & Friston,1995), and an estimate of intrinsic temporalautocorrelation was included within the model(Aguirre, Zarahn, & D’Esposito, 1997). Further-more, sine and cosine regressors for frequenciesbelow that of the task (0.0148 Hz) were alsoincluded in the general linear model. Temporaldata were smoothed with an empirically derivedestimate of the haemodynamic response of thefMRI system; this analysis has been empiricallydemonstrated to hold the mapwise false positiverate at or below tabular values (Zarahn et al.,1997).

RESULTS

Behavioural results

The task employed in this experiment was a directreplication of Solomon’s (1997) behavioural experi-ments. A summary of the relevant data from Solo-mon’s study is reported in Table 1 (for a full reportof the behavioural effects of these manipulations,please refer to Solomon’s study). In this experi-ment, we collected and analysed behavioural datafrom nine of our subjects as a manipulation check.Overall, we replicated the basic findings reportedby Solomon: Subjects were highly accurate at veri-fying properties for concepts and for verifyingletters in nonwords. Specifically, individuals in theassociated condition (M = 98.0%) were equallyaccurate at letter verification as subjects in the unas-

sociated condition (M = 97.8%). However, subjectsin the associated condition (M = 93.7%) were lessaccurate at verifying properties than subjects in theunassociated condition (M = 99.8%). In terms ofreaction times, on correct true trials, subjects in theassociated condition responded slightly slower onthe true trials (M = 829 ms) than subjects in theunassociated condition (M = 809 ms). Further-more, on correct false trials, subjects in the associ-ated condition responded slower on the associatedfalse trials (M = 885 ms) than subjects in the unas-sociated condition on the unassociated false trials(M = 833 ms).

fMRI results

The same analysis was performed on data fromboth the associated and the unassociated groups:To identify a significant property verification maineffect, all property verification trials were comparedto all baseline letter verification trials. In bothconditions, all subjects showed significant increasesin fMRI activity during the property verificationcondition relative to the baseline condition,mapwise α = .05 (see Figure 2). Two analyses wereperformed to examine group differences in thepattern of activation between subjects in the associ-ated and unassociated groups: First, we performed afocused region of interest analysis to test for groupdifferences in the specific area of visual cortexhypothesised to be involved in imagery and visualsemantics. Second, we conducted an exploratoryanalysis across the whole brain to identify othercandidate regions displaying group differences, ineither direction.

KAN ET AL.


Table 1. Summary of behavioural results (mean reaction times and mean error rates) from Solomon’s (1997) study and thecurrent study

Mean reaction times (SD) Mean error rate——————————————————– ——————————————————–

Solomon Current study Solomon Current study(n = 96) (n = 9) (n = 96) (n = 9)

————————— ———————— ————————— ————————Conditions True False True False True False True False

Associated 842 (182) 979 (276) 829 885 0.07 0.07 0.06 0.07Unassociated 731 (169) 748 (150) 809 833 0.05 0.02 0.01 0.01

ROI analyses. In each subject, an anatomicallydefined region of interest (ROI) in the left fusiformgyrus was created. Specifically, the local maximacoordinates for the left fusiform activity reported byD’Esposito et al. (1997) were used as a landmark(Talairach coordinates: –33, –48, –18). Subse-

quently, using those coordinates as the centre, asphere that encompassed 30 voxels (~2.11 cm3) wascreated. This ROI was then used to examine theeffects of property verification. The time seriesfrom all 30 voxels were averaged, and the effect sizefor property verification relative to baseline trials



a

b

Subject RB, ASSOCIATED condition

L R

Subject SF, UNASSOCIATED condition

L R

a

b

Figure 2. Data from representative subjects in (a) the associated condition and (b) the unassociated condition. Both subjects showedsignificant increases in fMRI activity during property verification condition relative to baseline condition (mapwise � = .05). However,only the associated subject showed significant activation in the left fusiform gyrus. Furthermore, neither subject showed significantactivation in the right fusiform gyrus. Anatomically defined regions of interest in the left and right fusiform gyri are indicated by thewhite outline boxes.

was calculated for the averaged time series for eachsubject. An unpaired t-test of these effect sizes,with subjects as a random variable and condition asa between-subject variable, was used to test forgroup differences. Furthermore, to examine poten-tial hemispheric differences, the same approach wasused to analyse activity within the right fusiformgyrus (Figure 2).

As a group, subjects in the associated conditionshowed significant activation in the left fusiformgyrus, M = 3.09; t(6) = 3.05, p < .05 (Figure 3). Onthe other hand, subjects in the unassociated groupdid not show significant activity in the sameanatomical region M = 0.26; t(6) = 0.451, p = .67

(Figure 3). An unpaired t-test revealed a significantdifference between the two groups, M = 2.83; t(12)= 2.44, p < .05.

To investigate laterality differences, a 2 (condi-tion) × 2 (hemispheres) mixed ANOVA wasconducted. A significant interaction was found,F(1, 12) = 4.80, p = .05. Probing with simple effectsrevealed a significant hemispheric difference in theassociated group, F(1, 6) = 10.27, p < .05, but not inthe unassociated group, F(1, 6) = 3.85, p = .10.

Whole-brain analyses. We performed an exploratorywhole-brain analysis that directly compared allvoxels in the associated and the unassociated condi-tions, using a random effects group analysis ofnormalised data from all subjects. Due to the smallnumber of subjects in each group, we did not haveenough power to detect above-threshold differ-ences using the conservative Bonferroni correctionfor ~18,000 voxels. We identified all voxels above athreshold of t = 3.5, based on an uncorrected alphalevel of .005. To guard against Type I error, weinitially included only activation with an extent ofseven or more contiguous voxels. Using thesecriteria, the only area found to be different betweenthe two conditions, with greater activation in theassociated condition than the unassociated condi-tion, was the left fusiform gyrus (see Figure 4). Thisregion was slightly superior (~10 mm) to thehypothesised region in the ROI analysis. When thecluster size was lowered to two contiguous voxels,the left middle frontal gyrus also revealed greateractivation for the associated condition. Further-more, right middle temporal gyrus showed greateractivation for the unassociated condition than theassociated condition (see Table 2).

KAN ET AL.


Table 2. Local maxima for areas of activation in the random effects group analysis of thewhole brain

Talairach coordinates ClusterRegion (Brodmann’s area) (x, y, z) size

at

Associated > UnassociatedFusiform gyrus (37) L –45 –53 –6 9 4.33Middle frontal gyrus (10) L –33 55 –6 2 4.05Unassociated > AssociatedMiddle temporal gyrus (21) R 48 –24 –6 2 3.81

aNo. of voxels.

-1

0

1

2

3

4

5

Av

erag

et-

val

ue

Regions of interest

Left Fusiform Gyrus Right Fusiform Gyrus

*

Associated

Unassociated

*

Figure 3. Average t-value of property verification minus baselineletter verification comparison within the left fusiform gyrus and theright fusiform gyrus.



Figure 4. Random effects group analysis of the whole brain. (Associated–Unassociated, cluster size = 7 contiguous voxels.)

DISCUSSION

In this study, we investigated the involvement ofvisual cortex during conceptual processing, in orderto test claims about the representation of semanticknowledge. Specifically, we investigated the extentto which the left occipito-temporal region of thebrain is involved in the property verification task.We examined a region of interest in the leftfusiform gyrus, which has been associated withmental imagery in previous studies (D’Esposito etal., 1997; Thompson-Schill et al., 1999). Wehypothesised that if semantic knowledge is indeedgrounded in the perceptual system, we should findsignificant activity in the left fusiform gyrus duringproperty verification, but only when conceptualinformation is required (i.e., when associated falsetrials were present). Two aspects of the results ofthis study are noteworthy: The location of activa-tion associated with the property verification taskand the conditions under which this activation wasobserved. Each of these points is elaborated below.

The first important finding of this study was thatretrieval of semantic knowledge, in the absence ofexplicit mental imagery instructions, activated aregion of visual association cortex that has beenassociated with visual object recognition (e.g.,Ishai, Ungerleider, Martin, & Haxby, 2000; for areview, see Kanwisher, Downing, Epstein, &Kourtzi, 2001) and visual imagery (e.g., D’Espositoet al., 1997). Activation of visual association cortexduring semantic processing supports the idea thatconceptual knowledge is grounded in the percep-tual system. Furthermore, we replicated the hemi-spheric asymmetry reported in other studies onmental imagery (e.g., D’Esposito et al., 1997;Farah, 1984; Riddoch, 1990). The absence of acti-vation in the right fusiform gyrus serves as a controlregion to show that effects reported here are specificto areas involved in mental imagery rather thannonspecific increases in activation related to taskdifficulty or duration of processing. In addition, thehemispheric differences may have implications forunderstanding the processes involved in retrieval ofsemantic knowledge. Farah (1995) has claimed thatthe process of mental image formation is lateralisedto the left hemisphere, despite the fact that visual

perception occurs, at least in its initial stages,symmetrically in both hemispheres. Barsalou(1999) argued that retrieval of conceptual knowl-edge requires the simulation of perceptual informa-tion; our results suggest that the process of visualsimulation during semantic retrieval is, like mentalimagery, a left-hemisphere lateralised process.Differential involvement of the left and rightfusiform gyri has also been reported in studies ofobject recognition. In an fMRI experiment using arepetition priming paradigm, Koutstaal, Wagner,Rotte, Maril, Buckner, and Schacter (2001)reported differential neural effects of priming in leftand right fusiform gyri. Specifically, it was reportedthat visual form changes between first and secondexposures were more “costly” in the right fusiformthan in the left fusiform. That is, perceptualchanges of stimuli led to a greater reduction ofneural priming effects in the right fusiform than theleft fusiform. Future studies may help elucidate thenature of hemispheric specialisation in the fusiformgyrus, as it relates to both image formation andobject recognition.

The second important finding was that activa-tion of this region only occurred under conditionshypothesised to require conceptual knowledge.Within the left fusiform gyrus, we found reliableactivity across subjects when true trials were inter-mixed with associated false trials, but not when truetrials were intermixed with unassociated false trials.This pattern of activation follows from the taskanalysis provided by Solomon (1997). Based onpatterns of response times, she argued that twoprocesses are available to perform the property veri-fication task: First, one can consult informationabout lexical co-occurrence, or strength of associa-tion between two words; this process will only beeffective when associative strength is a reliable indi-cator of the correct response (i.e., in the unassoci-ated condition). Second, one can retrieve semanticinformation about the properties of an object byperceptually simulating the representation; thisprocess is assumed to be less efficient than theformer, and will thus be preferred only when theother process is ineffective (i.e., in the associatedcondition). According to this framework, differ-ences in activation between the associated and


KAN ET AL.

unassociated condition will reflect the semanticretrieval and perceptual simulation demands thatare unique to the associated condition. Aspredicted, these differences were observed in visualassociation cortex.

It is possible that activation in other regions ofcortex would also differ between the two condi-tions, either as a result of an extended network ofcortex that supports perceptual simulation or as aconsequence of other cognitive differences betweenthe two tasks (e.g., conflict monitoring in the moredifficult, associated condition). In order to evaluateother, unanticipated differences, we conducted anexploratory analysis of effects across the entirebrain. With a cluster size of seven contiguousvoxels, the left fusiform gyrus was the only regionthat revealed greater activity in the associated thanthe unassociated condition. This result is consistentwith the region of interest analysis reported above,although the location of the cluster is slightlysuperior to the a priori region of interest based onthe results of D’Esposito et al. (1997). Althoughthe results of the whole-brain analysis must beinterpreted with caution because a conservativethreshold was not applied, the trend in the dataindicates that the left fusiform gyrus is selectivelyrecruited by whatever processing differences existbetween the two conditions. Based on previousresearch, differences in the left fusiform gyrus arelikely to reflect perceptual simulation (or imageformation) that accompanies semantic knowledgeretrieval.

The task analysis described above would seem tosupport the prediction that regions associated withretrieval of lexical associations would be more activein the unassociated condition relative to the associ-ated condition. In the unassociated minus associ-ated contrast, the only area that showed a hint ofactivation was in the right middle temporal gyrus.However, note that this activation is present onlywhen the cluster threshold is lowered to two contig-uous voxels (see Table 2), so it would be prematureto draw any strong inferences based on this result.One possible reason for the lack of greater activa-tion in the unassociated condition may be due toautomatic word association processing, which iscommon to both conditions. According to the

priming literature, word association is assumed tobe a relatively automatic process (e.g., Meyer &Schvaneveldt, 1971; Shelton & Martin, 1992).When subjects perform the task, word associationprocesses may be triggered automatically. Thus,brain areas that are recruited for word associationswould be activated for subjects in both associatedand unassociated conditions, and a direct compar-ison would reveal no differences in relevant brainregions. Another possible explanation for the lackof increased activation in the unassociated condi-tion is that perhaps there is more between-subjectvariability in the patterns of activation associatedwith retrieval of linguistic information. Since agroup analysis was performed, only areas that arereliably activated for all subjects were identified.Therefore, increased between-subject variabilityassociated with a given process would result infewer common areas found to be active in the groupanalysis.

Thus far, we have illustrated how one manipu-lation (i.e., relationship between true and falsetrials in a property verification task) modulatesactivity in the visual association cortex. It isreasonable to further examine how other factorsmay affect activity in this part of the brain. Datafrom neuroimaging (e.g., Martin, Wiggs,Ungerleider, & Haxby, 1996; Mummery,Patterson, Hodges, & Wise, 1996; Thompson-Schill et al., 1999) and neuropsychological studies(e.g., Farah, McMullen, & Meyer, 1991;Warrington & McCarthy, 1987; Warrington &Shallice, 1984) have indicated that there may becategory-specific effects in the retrieval ofsemantic knowledge. It is, then, reasonable tosuppose that the effects that we have observed inthe current study may differ across different typesof stimuli. For example, perhaps the processesinvolved in verifying properties of living thingsmay differ from verifying properties of nonlivingthings. Unfortunately, since a blocked design wasused, we are unable to address that issue in thisstudy. Thus, it will be informative to investigatethese effects in an imaging study using an event-related design.

For the past few decades, Pylyshyn has main-tained that subjects in mental imagery experiments



are merely simulating the use of visual spatial repre-sentations, even though the internal representa-tions are organised amodally (for a review, seePylyshyn, 1981). He further argued that this isparticularly true in experiments that explicitly asksubjects to imagine a certain object or scenario. Hepostulated that the subjective experience of “seeingwith the mind’s eye” is purely epiphenomenal.Researchers who support the idea that conceptualknowledge is grounded perceptually have tried touse neuroimaging data to counter Pylyshyn’s argu-ment. For example, in a review chapter, Kosslynand Thompson (2000) concluded that visualperception and mental imagery share manycommon neural mechanisms on the basis of manyneuroimaging studies. Although it is difficult toimagine a way in which subjects could “simulate”blood flow to certain regions of the brain, Pylyshyn(in press) has recently extended his criticisms to thefunctional neuroimaging literature as well.

We believe that our current study addresses partof Pylyshyn’s concern. In the present experiment,although subjects were told to verify each propertywith the corresponding concept, they were notgiven explicit imagery (or nonimagery) instructionson how to perform the task. Thus, it seems improb-able that subjects would interpret the task asrequiring them to simulate the use of visual repre-sentations. Furthermore, it is even less plausiblethat all of the associated subjects would choose tosimulate the use of visual spatial representationsand that the unassociated subjects would choosenot to simulate that experience. Our finding thatmanipulating the type of false trials modulates acti-vation of the visual association areas presents astrong argument against Pylyshyn’s (in press)claims. If neural activity is purely epiphenomenal,there is no reason to expect the group differencesobserved in this experiment. It is difficult to explainthe group differences observed in this study withPylyshyn’s reasoning. In contrast, our results areeasily explained by differences between the twoconditions with regard to requirements for concep-tual knowledge, which is grounded in the percep-tual system.

REFERENCES

Aguirre, G. K., Zarahn, E., & D’Esposito, M. (1997).Empirical analyses of BOLD fMRI statistics. II.Spatially smoothed data collected under null-hypoth-esis and experimental conditions. NeuroImage, 5,199–212.

Allport, D. A. (1985). Distributed memory, modularsubsystems and dysphasia. In S. K. Newman & R.Epstein (Eds.), Current perspectives in dysphasia (pp.207–244). Edinburgh: Churchill Livingstone.

Avotec. (2002). Retrieved February 8, 2002, from http://www.avotec.org

Barsalou, L. W. (1999). Perceptual symbol systems.Behavioral and Brain Sciences, 22, 577–660.

Barsalou, L. W., Solomon, K. O., & Wu, L. L. (1999).Perceptual simulation in conceptual tasks. Paperpresented at the Proceedings of the 4th conference ofthe International Cognitive Linguistics Association,Amsterdam, The Netherlands.

Bisiach, E., & Luzzatti, C. (1978). Unilateral neglect ofrepresentational space. Cortex, 14, 129–133.

Charlot, V., Tzourio, M., Zilbovicius, M., Mazoyer,B., & Denis, M. (1992). Different mental imageryabilities result in different regional cerebral bloodflow activation patterns during cognitive tasks.Neuropsychologia, 30, 565–580.

Cohen, J. D., MacWhinney, B., Flatt, M., & Provost, J.(1993). PsyScope: A new graphic interactive environ-ment for designing psychology experiments. Behav-ioral Research Methods, Instruments, and Computers,25, 257–271.

D’Esposito, M., Detre, J. A., Aguirre, G. K., Stallcup,D., Alsop, D. C., Tippett, L. J., & Farah, M. J.(1997). A functional MRI study of mental imagegeneration. Neuropsychologia, 35, 725–730.

Evans, A. C., Collins, D. L., Mills, S. R., Brown, E. D.,Kelly, R. L., & Peters, T. M. (1993). 3D statisticalneuroanatomical models from 305 MRI volumes. Paperpresented at the IEEE-Nuclear Science Symposiumand Medical Imaging Conference.

Farah, M. J. (1984). The neurological basis of mentalimagery: A componential analysis. Cognition, 18,245–272.

Farah, M. J. (1995). Current issues in the neuro-psychology of image generation. Neuropsychologia, 33,1455–1471.

Farah, M. J., McMullen, P. A., & Meyer, M. M. (1991).Can recognition of living things be selectivelyimpaired? Neuropsychologia, 29, 185–193.

KAN ET AL.


Farah, M. J., Soso, M. J., & Dasheiff, R. M. (1992).Visual angle of the mind’s eye before and after unilat-eral occipital lobectomy. Journal of ExperimentalPsychology: Human Perception and Performance, 18,241–246.

Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionismand cognitive architecture: A critical analysis. Cogni-tion, 28, 3–71.

Gentner, D. (1978). On relational meaning: The acquisi-tion of verb meaning. Child Development, 49, 988-998.

Glaser, W. R. (1992). Picture naming. Cognition, 42, 61–106.

Goldenberg, G., Podreka, I., Steiner, M., Franzen, P., &Deecke, L. (1991). Contributions of occipital andtemporal brain regions to visual and acoustic imag-ery—a SPECT study. Neuropsychologia, 29, 695–702.

Goldenberg, G., Podreka, I., Steiner, M., & Willmes, K.(1987). Patterns of regional cerebral blood flowrelated to memorizing of high and low imagery words:An emission computer tomography study.Neuropsychologia, 25, 473–485.

Intons-Peterson, M. J. (1983). Imagery paradigms: Howvulnerable are they to experimenters’ expectations?Journal of Experimental Psychology: Human Perceptionand Performance, 9, 394–412.

Ishai, A., Ungerleider, L. G., Martin, A., & Haxby, J. V.(2000). The representation of objects in the humanoccipital and temporal cortex. Journal of CognitiveNeuroscience, 12(suppl. 2), 35–51.

Kanwisher, N., Downing, P., Epstein, R., & Kourtzi, Z.(2001). Functional neuroimaging of human visualrecognition. In R. Cabeza & A. Kingstone (Eds.),The handbook of functional neuroimaging (pp. 109–152). Cambridge, MA: MIT Press.

Kosslyn, S. M. (1975). Information representation invisual images. Cognitive Psychology, 7, 341–370.

Kosslyn, S. M. (1976). Can imagery be distinguishedfrom other forms of internal representation? Evidencefrom studies of information retrieval time. Memoryand Cognition, 4, 291–297.

Kosslyn, S. M. (1994). Image and brain. Cambridge,MA: MIT Press.

Kosslyn, S. M., Ball, T. M., & Reiser, B. J. (1978). Visualimages preserve metric spatial information: Evidencefrom studies of image scanning. Journal of Experimen-tal Psychology: Human Perception and Performance, 4,47–60.

Kosslyn, S. M., & Thompson, W. L. (2000). Sharedmechanisms in visual imagery and visual perception:

Insights from cognitive neuroscience. In M. S.Gazzaniga (Ed.), The new cognitive neurosciences.Cambridge, MA: MIT Press.

Koutstaal, W., Wagner, A. D., Rotte, M., Maril, A.,Buckner, R. L., & Schacter, D. L. (2001). Perceptualspecificity in visual object priming: Functionalmagnetic resonance imaging evidence for a lateralitydifference in fusiform cortex. Neuropsychologia, 39,184–199.

Locke, J. (1959). An essay concerning human understand-ing, Vol. 1 & 2 (1st ed.) (original work published in1960). New York: Dover.

Martin, A., Wiggs, C. L., Ungerleider, L. G., & Haxby,J. V. (1996). Neural correlates of category-specificknowledge. Nature, 379, 649–651.

Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilita-tion in recognizing pairs of words: Evidence of adependence between retrieval operations. Journal ofExperimental Psychology, 90, 227–234.

Mummery, C. J., Patterson, K., Hodges, J. R., & Wise,R. J. (1996). Generating “tiger” as an animal name or aword beginning with T: Differences in brain activa-tion [published erratum appears in Proc R Soc LondB Biol Sci 1996 Dec 22;263(1377):1755–6]. Proceed-ings of the Royal Society of London - Series B: BiologicalSciences, 263, 989–995.

Nelson, K. (1974). Concept, word, and sentence: Inter-relations in acquisition and development. Psychologi-cal Review, 81, 267-285.

Price, H. H. (1953). Thinking and experience. London:Hutchinson’s Universal Library.

Pylyshyn, Z. W. (1981). The imagery debate: Analoguemedia versus tacit knowledge. Psychological Review,88, 16–45.

Pylyshyn, Z. W. (1984). Computation and cognition:Toward a foundation for cognitive science. Cambridge,MA: MIT Press.

Pylyshyn, Z. W. (in press). Mental imagery: In search of atheory. Behavioral and Brain Sciences.

Riddoch, M. J. (1990). Loss of visual imagery: A genera-tion deficit. Cognitive Neuropsychology, 7, 249–273.

Roland, P. E., & Friberg, L. (1985). Localization ofcortical areas activated by thinking. Journal ofNeurophysiology, 53, 1219–1243.

Rosch, E., Mervis, C. B., Gray, W., Johnson, D., &Boyes-Braem, P. (1976). Basic objects in natural cate-gories. Cognitive Psychology, 8, 382–439.

Servos, P., & Goodale, M. A. (1995). Preserved visualimagery in visual form agnosia. Neuropsychologia, 33,1383–1394.



Shelton, J. R., & Martin, R. C. (1992). How semantic isautomatic semantic priming? Journal of ExperimentalPsychology: Learning, Memory, and Cognition, 18,1191–1210.

Shepard, R. N., & Cooper, L. A. (1982). Mental imagesand their transformations. New York: CambridgeUniversity Press.

Solomon, K. O. (1997). The spontaneous use of perceptualrepresentations during conceptual processing. Chicago:University of Chicago.

Thompson-Schill, S. L. (2003). Neuroimaging studies ofsemantic memory: Inferring “how” from “where.”Neuropsychologia, 43, 280–292.

Thompson-Schill, S. L., Aguirre, G. K., D’Esposito, M.,& Farah, M. J. (1999). A neural basis for category andmodality specificity in semantic knowledge.Neuropsychologia, 37, 671–676.

VoxBo. (2002). Retrieved February 8, 2002, from http://www.voxbo.org

Warrington, E. K., & McCarthy, R. (1987). Categoriesof knowledge: Further fractionations and anattempted integration. Brain, 110, 1273–1296.

Warrington, E. K., & Shallice, T. (1984). Categoryspecific semantic impairments. Brain, 107, 829–854.

Worsley, K. J., & Friston, K. (1995). Analysis of fMRItime-series revisited—again. NeuroImage, 2, 173–182.

Zarahn, E., Aguirre, G. K., & D’Esposito, M. (1997).Empirical analyses of BOLD fMRI statistics. I.Spatially unsmoothed data collected under null-hypothesis conditions. NeuroImage, 5, 179–197.

Zatorre, R. J., Halpern, A. R., Perry, D. W., Meyer, E.,& Evans, A. C. (1996). Hearing in the mind’s ear: APET investigation of musical imagery and percep-tion. Journal of Cognitive Neuroscience, 8, 29–46.

KAN ET AL.


Role of mental imagery in a property verification task

Documents

Transcript of Role of mental imagery in a property verification task