Can We Simulate Negation?The Simulation Effects of Negation in English Intransitive Sentences

Meylysa TsengJung-Hee KimBenjamin BergenUniversity of Hawai’i at ManoaBLS 2006

2

Thanks to

• Amy Schafer

• William O’Grady

• Kathryn Wheeler

• Cognitive Linguistics

Research Group

at University of Hawaii

3

Mental imagery

• When you hear The dolphin soared,

You may be mentally picturing (simulating) a visual scene

• Mental imagery comes from our perceptual or motor experience

• Mental imagery should resemble our perceptual or motor experience

4

Imagery in Language

• Imagery is crucial in language use (Zwaan 1999, Barsalou 1999, Glenberg & Robertson

2000, Bergen & Chang 2005)

• The idea: – Language is associated with concrete perceptual or

motor experiences

– These experiences are recreated through imagery during language understanding

– Such perceptual or motor imagery is activated in language understanding• The dolphin soared (visual imagery)• I pushed the door (motor imagery)

5

Dimensions of Visual imagery used in language processing

• Orientation of objects (Stanfield & Zwaan 2001)

• Shape of objects (Zwaan et al. 2002)

• Spatial location (Richardson et al. 2003, Lindsay 2003)

• Upwards or downwards motion (Bergen 2005)

6

Bergen 2005: Up or Down visual imagery

• What parts of the sentences yield the simulation effects?– The mule climbed (Up-Verb)– The glass fell (Down-Verb)– The sky darkened (Up-Noun)– The ground shook (Down-Noun)

• It was concrete Subject Nouns and Verbs denoting up/down motion or location

• Understanding sentences with subject nouns or verbs that entail upness or downness visual imagery selectively interfered with the visual processing of objects in the same parts of the visual field

7

Today: Simulating Negation

• Negation presents an especially difficult case for a simulation-based approach

• You can’t imagine what’s not happening

• How can negation be captured or represented in a simulation-based account?

8

Questions remain

• Understanding negated sentences also engages mental simulation?

• Up/down visual imagery is activated in comprehending ‘The mule didn’t climb’ or ‘The sky didn’t darken’?

9

Simulation-based account of negation comprehension

• Negation can be understood as a process

“Negation is implicitly encoded in the simulation processes that are undertaken when comprehending a negative sentence”

(Kaup and Lüdtke 2005)

• Negation involves two simulations– Simulation of the counterfactual situation– Suppression– Simulation of the factual situation

10

This study: Negation & Up/Down Visual Imagery

• This study replicated Bergen 2005

• We hypothesize: mental simulation is engaged in understanding negation

• We test: Up/Down visual imagery is activated by such up/down-associated verbs and nouns even when embedded in the negated sentences

• Stimuli: up/down-associated negative sentences

– We negated Bergen’s literal sentences with up or down-associated nouns or verbs

The mule didn’t climb (Up-Verb)The glass didn’t fall (Down-Verb)

The sky didn’t darken (Up-Noun) The ground didn’t shake (Down-Noun)

11

Predictions

(1) In processing negated sentences, if up/down associated noun and verbs induce up/down visual imagery especially in the counterfactual simulation stage, there should be an interaction effect between noun/verb location and object location in a visual object categorization task

(2) Consistent with the proposed stages of negation simulation, we would see a shift of simulation from counterfactual to factual over time

12

Time course – 200ms, 500ms

• 200ms – the initial counterfactual simulation– Up/Down visual imagery was activated at 200ms time delay in

processing up/down-associated ‘positive’ sentences (Bergen 2005)

• 500ms – the later factual simulation – ‘Negation seemed to effect some time between 500 and 1,000

msec from the time the negative sentence had been read’ (Hasson and Gluksberg 2004:20)

• Prediction: – in processing up/down-associated ‘negative’ sentences,

up/down visual imagery would be activated at 200ms and there would be a change in the simulation at 500ms

13

Method: replicate Bergen 2005• fixation cross (1s)

• Ss heard a sentence

• Either 200 ms or 500 ms delay

• Ss saw a circle or square on the screen: up, down, right, or left (for 200 msec)

– Critical trials: up or down.

– The same number of (non-critical) sentences followed by an object on the left or right

• Ss categorize visual object

• Subjects press circle (z) or square (x)

• A comprehension question was interleaved, to ensure listening for content

14

Example

15

+++

16

“The mule didn’t climb.”

(heard over the headphone)

17

18

+++

19

“The glass didn’t fall.”

(heard through headphone)

20

21

Result: Negated Verb Sentences

• Up: The mule didn’t climb

• Down: The glass didn’t fall

22

400

440

480

520

560

600

640

680

720

DownVerb

UpVerb

DownVerb

UpVerb

DownNoun

UpNoun

DownNoun

UpNoun

200 ms 500 ms 200 ms 500 ms

Rea

ctio

n Ti

me

(ms)

SameDifferent

Negated Verb Sentences (200 ms)

*

23

400

440

480

520

560

600

640

680

720

DownVerb

UpVerb

DownVerb

UpVerb

DownNoun

UpNoun

DownNoun

UpNoun

200 ms 500 ms 200 ms 500 ms

Reac

tion

Tim

e (m

s)

SameDifferent

Negated Verb Sentences (500 ms)

*

24

Result: Negated Noun Sentences

• Up: The rainbow didn’t fade

• Down: The ground didn’t shake

• “The sky didn’t darken” was eliminated due to having average means higher than 2.5 SD from the mean

25

400

440

480

520

560

600

640

680

720

DownVerb

UpVerb

DownVerb

UpVerb

DownNoun

UpNoun

DownNoun

UpNoun

200 ms 500 ms 200 ms 500 ms

Rea

ctio

n Ti

me

(ms)

SameDifferent

Negated Noun Sentences (200 ms)

*

*

26

400

440

480

520

560

600

640

680

720

DownVerb

UpVerb

DownVerb

UpVerb

DownNoun

UpNoun

DownNoun

UpNoun

200 ms 500 ms 200 ms 500 ms

Rea

ctio

n Ti

me

(ms)

SameDifferent

Negated Noun Sentences (500 ms)

*

* *

27

400

440

480

520

560

600

640

680

720

DownVerb

UpVerb

DownVerb

UpVerb

DownNoun

UpNoun

DownNoun

UpNoun

200 ms 500 ms 200 ms 500 ms

Rea

ctio

n Ti

me

(ms)

SameDifferent

Summary of Result

*

* *

28

Summary of Effect

Verb

200ms Significant interaction

500ms Marginal interaction

Noun

200ms

Significant interaction 500ms

- Responses were faster when the sentence location

coincided with the visual object location

29

Findings: Prediction 1

• There was a significant interaction effect between up/down noun/verb location and object location

• Up/down nouns or verbs induced visual simulation even when embedded in negated sentences

• This thus supports our main hypothesis that negation induces mental simulation

30

Findings: Prediction 2

• Same interaction effect at both 200ms and 500ms, which results from the counterfactual simulation

• We found an evidence for a simulation of the counterfactual situation shortly after processing of negative sentences

• We found no evidence for factual simulation• We found no evidence of a shift in the

simulation in negation processing

31

No evidence for factual simulation

• If the glass didn’t fall,…

• We didn’t give clear factual situation

• Without context, factual situation is relatively underspecified

• This lead to not clear factual simulation stage

32

Bergen 2005 vs This study

interference

Positive

facilitation500 ms

facilitation200 ms

Negative

• These two studies showed that up/down visual imagery is activated in processing negated sentences as well as positive sentences with up/down nouns or verbs

• Interestingly, our interaction was in the opposite direction.

33

Bergen 2005 vs. This study(Interference vs. Facilitation)

400440480520560600640680720

Dow

n Up

Dow

n Up

Dow

n Up

Dow

n Up

Dow

n Up

Dow

n Up

LiteralVerb

Noun LiteralVerb

LiteralVerb

Noun Noun

200ms 200ms 200ms 500ms 200ms 500ms

Bergen 2005 This study

Res

pon

se T

ime

(ms)

SameDifferent

** ** *

34

Discussion: Interference vs. Facilitation

• Interference and facilitation effects in imagery studies are both simulation effects.

• The two effects basically result from the use of the same neural structures to understand language and perform a perception

• Two factors (Kaschak et al. 2004)– Extent of Time overlap between the two

processing involved (sentence processing and visual processing)

– Extent of Integratability of the two tasks “the extent to which the perceptual stimulus (circle or

square) can be integrated into the simulation constructed of the content of the sentence”

35

How the two factors work

• If sentence processing and visual processing are,– Sequential – Facilitation as a form of semantic priming– Simultaneous – Interference or Facilitation

depending on Integratability Integratable – Facilitation Non-integratable – Interference

• Interference – perform two different (non-integratable) tasks at the same time• Facilitation – perform two related (integratable) tasks at the same time

36

Interference in Bergen’s positive sentences

• Simultaneous processing (200ms time overlap)• Less integratability in simulating positive sentences

– The sky darkened– Positive sentence tells us exactly what happened. It

is fully informative about the location to be simulated

– Such detailed image construction is not readily integratable with a circle or a square visual processing

– Such specifically simulated image blocks visual processing

– Therefore, interference effect

37

Facilitation in our negative sentences

• Simultaneous processing (200ms overlap)• Relatively more integratability involved in simulating

negative sentence

- The sky didn’t darken- A single negative sentence with no context is less

informative (less specific or less detailed) in its conveyed meaning

- Perceivers would simulate less well-defined location imagery and thus construct a less-detailed simulation

- This less-detailed image construction and a circle or a square visual processing can be relatively integratable

- Thus, this less-detailed simulated image doesn’t block visual processing

- Therefore, facilitation effect

38

Summary of discussion

• Positive sentence processing – Fully informative – Well-defined and detailed image construction– Low integratability– Interference

• Negative sentence processing– Less informative– Less well-defined and less-detailed image

construction– Makes the two tasks relatively integratable– Facilitation

39

400

440

480

520

560

600

640

680

720

DownVerb

UpVerb

DownVerb

UpVerb

DownNoun

UpNoun

DownNoun

UpNoun

200 ms 500 ms 200 ms 500 ms

Rea

ctio

n Ti

me

(ms)

SameDifferent

Discussion: Verb effects significant only at 200ms

*

* *

40

Weaker simulation effect for verbs at 500ms

• Scope of negation– The sky [didn’t darken] (Up-noun)– The glass [didn’t fall] (Down-verb)

• For nouns, the direction is not changed under negation: less subject variability

• For verbs, the direction is changed under negation: more subject variability

• This would make verbs exhibit relatively weaker simulation at 500ms

41

More variability in Verbs (200ms)

N O U N at 200m s

400440480520560600640680720760800

1 3 5 7 9

Se n te n c e (1 -5 D o w n , 6 -9 U p )

RT (m

s)

S am e

D iffe re n tVER B at 200m s

400440480520560600640680720760800

1 3 5 7 9

Se n te n c e (1 -5 D o w n , 6 -1 0 U p )

RT (m

s)

S am e

D iffe re n t

42

Summary• Found an interaction effect of noun/verb direction with

visual object recognition in comprehending negated intransitive sentences

• Negated sentences involve a less detailed simulation, causing facilitation

• Imagery is involved in the comprehension of negated sentences

• Found an evidence for a simulation of the counterfactual situation

• Found no evidence for factual simulation• Found no evidence for a shift in the simulation of

negation

43

Future Study

• How to capture a shift in the process of negation simulation?

• To capture the further clear counterfactual simulation, test sentences at an earlier time course (0ms)

• To capture the clear factual simulation, give a clear factual situation, since without context, the factual situation is relatively underspecified– The glass didn’t fall. It rose.

44

Conclusions

• Spatial imagery of varying degrees of detail is automatically or unconsciously engaged in language understanding

• Language about a visual scene induces visual simulation, even in negated sentences

• How we actually process negated sentences crucially depends upon the detailed semantics of sentential constituents, such as subject nouns and verbs

• Embodied view of meaning

– “[…] the core of our conceptual systems is directly grounded in perception, body movement, and experience of a physical and social character.” (Lakoff 1987, p.xiv)

45

Thank you

46

References• BARSALOU, LAWRENCE W. 1999. Perceptual symbol systems. Behavioral and Brain Sciences, 22.577-660.• BERGEN, BENJAMIN K., NARAYAN, SHWETA and FELDMAN, JEROME. 2003. Embodied verbal semantics:

evidence from an image-verb matching task. Paper presented at Cognitive Science Conference. .• GLENBERG, ARTHUR M. and KASCHAK, MICHAEL P. 2002. Grounding Language in Action. Psychonomic Bulletin

& Review, 9.558-65.• Hasson, Uri and Sam Glucksberg. 2004. Does Understanding Negation Entail Affirmation? An Examination of • Negated Metaphors. manuscript. University of Chicago.• Just, M. A., and P. A. Carpenter. 1976. Eye fixations and cognitive processes. Cognitive Psychology, 8: 441-480.• Kaschak, Michael P, Rolf A. Zwaan, Mary Aveyard and Richard H. Yaxley. (in press). Perception of auditory motion

affects language processing. Cognitive Science.• Kaschak, Michael P., Carol J. Madden, David J. Therriault, Richard H. Yaxley, Mark Aveyard, Adrienne A. Blanchard &

Rolf A. Zwaan. 2004. Perception of motion affects language processing. Cognition 94:B79-B89. • Kaup, Barbara. 2001 Negation and its impact on the accessibility of text information. Memory & Cognition, 29, 860-967.• KAUP, BARBARA and ZWANN, ROLF A. 2003. Effects of negation and situational presence on the accessibility of text

information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29.439-46.• Lindsay, Shane (2003) Visual priming of language comprehension. Masters Thesis. Department of Cognitive and

Computing Sciences. University of Susses.• MacDonald, Maryellen C and Just, Marcel A. 1989. Changes in activation levels with negation. Journal of Experimental

Psychology: Learning, Memory and Cognition. Jul;15(4):633-42.• NARAYAN, SHWETA, BERGEN, BENJAMIN K. and WEINBERG, ZACH. 2004. Embodied verbal semantics:

Evidence from a lexical matching task. Paper presented at The Thirtieth Berkeley Linguistics Society Annual Meeting.• RICHARDSON, DANIEL C., SPIVEY, MICHAEL J., BARSALOU, LAWRENCE W. and MCRAE, KEN. 2003. Spatial

representations activated during real-time comprehension of verbs. Cognitive Science.767-80.• ZWANN, ROLF A., STANFIELD, ROBERT A. and YAXLEY, RICHARD H. 2002. Language comprehenders mentally

represent the shapes of objects. Psychological Science, 13.168-71.• ZWANN, ROLF A. 1999. Embodied cognition, perceptual symbols, and situation models. Discourse Processes, 28.81-88.

47

48

Stimuli

• Down-Verb– The stone didn’t sink– The chair didn’t topple– The pipe didn’t drop– The glass didn’t fall– The cat didn’t descend

• Up-Verb– The mule didn’t climb– The lizard didn’t ascend– The cork didn’t rocket– The patient didn’t rise– The dolphin didn’t soar

• Down-Noun– The grass didn’t glisten– The submarine didn’t fire– The cellar didn’t flood– The ground didn’t shake– The shoe didn’t smell

• Up-Noun– The sky didn’t darken– The rainbow didn’t fade– The roof didn’t creak– The tree didn’t sway– The ceiling didn’t crack

49

NOUN RTs

　 　 Same Different

The cellar didn’t flood. NounDown 631.00 620.42

The grass didn’t glisten. NounDown 659.68 596.39

The ground didn’t shake. NounDown 592.68 535.95

The shoe didn’t smell. NounDown 577.58 687.50

The submarine didn’t fire. NounDown 589.33 657.51

The ceiling didn’t crack. NounUp 615.32 718.55

The rainbow didn’t fade. NounUp 498.11 704.59

The roof didn’t creak. NounUp 475.05 644.83

The tree didn’t sway. NounUp 482.95 750.37

50

VERB RTs

　 　 Same Different

The cat didn’t descend. LiteralDown 602.91 660.43

The chair didn’t topple. LiteralDown 516.69 512.38

The glass didn’t fall. LiteralDown 477.27 605.47

The pipe didn’t drop. LiteralDown 467.45 616.74

The stone didn’t sink. LiteralDown 571.09 477.73

The cork didn’t rocket. LiteralUp 507.00 451.55

The dolphin didn’t soar. LiteralUp 452.67 475.00

The lizard didn’t ascend. LiteralUp 607.55 590.51

The mule didn’t climb. LiteralUp 444.95 613.73

The patient didn’t rise. LiteralUp 515.73 502.09

51

Bergen’s findings interaction effect (interference) between

noun/verb location and object location

400440480520560600640680720

Down Up Down Up

Literal Verb Noun

200ms 200ms

Resp

onse

Tim

e (

ms)

SameDifferent

* *