Embodied Language, Cognition, and Communication (E LCC): Spontaneous gesture with

speech

Alan Cienki and Irene MittelbergEngelse taalkunde

Afdeling Taal en CommunicatieFaculteit der Letteren

LCC presentation, 6 November 2008

Some background: gesture studies

• Spoken language studied in fields such as anthropology, sociology (e.g., CA), ethnography of communication

• Gesture sometimes studied in these contexts

• The study of “non-verbal communication” in clinical psychology

Gesture studies as its own field

• E.g., Adam Kendon’s work since 1970s– Gesture: Visible action as utterance (2004)

• Landmark – Hand and mind: What gestures reveal about thought (David McNeill 1992)

• International Society for Gesture Studies (ISGS)• ISGS conference series• Founding of the journal Gesture in 2001

What are we calling a gesture?

• A visible, distinct, effortful movementof part of the body (Kendon)

• Special status of manual gestures

Background on gesture

• Range of conventionality of form-meaning associations –

from highly idiosyncratic to highly conventionalized

“Kendon’s continuum” (Kendon 1982; McNeill 1992)

gesticulation - pantomime - emblems - sign language

Gesturing is not usually just handwaving

• Not random, structureless or unsystematic

• Basic structuring principles of gesture at work:– schematic patterns– abstractions from physical actions, objects,

and relations– roles of metonymy and metaphor

On the coding of gestural forms

• Four parameters are often used (based on research on sign languages):– hand shape– palm orientation– location in gesture space– movement

Notation of form features in gestures

• status of fingers differentiated

(Bressem 2006, in prep.)

Notation of form features in gestures

• form class “flat hand”

(Bressem 2006, in prep.)

Models of gesture space

1

32

0

left

right

upper

lower

extreme peripheryperiphery

center cc

-1

etc.

(Fricke 2005/ in prep.) (McNeill 1992)

Some functions of gestures

• Discourse structuring function

• Pragmatic/interactional function

• Referential function

• These can overlap– Gestures can have multiple functions

Some functions of gestures

• Referential function– Concrete reference

– Abstract reference

Some functions of gestures

Concrete reference

Oh, an essay.You have to `write fast!

{video clip}

(video clips have not been included in this PowerPoint since we do not have permission to publish them)

Some functions of gestures

• Abstract reference:Representation of the abstract (non-physical) in terms of the concrete e.g., an idea as a space, form, or motion

Metaphor

Example: metaphors in gesturesthat,there’s never a situation, that is,ideally r- -- where there’s,an ideal,<@well@>, <at least there isn’t in my life>,where there’s something right, and something absolutely wrong.What you have to do is draw your line,and figure out on which side of it you fall.

Example: metaphors in gestures

{video clip}

Example: metaphors in gestures

Possible analysis might include:• SITUATIONS ARE LOCATIONS

• MAKING A DECISION IS DRAWING A LINE

• EVALUATING YOUR BEHAVIOR IS POSITIONING YOURSELF

• Note timing of onset of gestures with respect to speech

Metaphor

• Metaphor as not just a matter of language, but of mapping one conceptual domain (with imagery!) onto another (Lakoff & Johnson 1980, 1999; and many other works…)

• Therefore, we should see evidence of conceptual metaphors in different forms of human expression

• Potential for multimodal expression of metaphor in language use (Cienki & Müller 2008; Forceville & Urios-Aparisi, forthcoming; Müller 2008)

Side note: perspective in gesture studies

• If studying gesture can help in meaning-making, whose meaning-making are we talking about?– the speaker’s?– the addressee’s?

– another observer’s? (the researcher’s?)

Interaction of metaphor and metonymy in co-verbal gestures

Irene Mittelberg

points of departure:

How do speech and gesture share the semiotic work of rendering abstract knowledge domains

more graspable?

Previous research has shown that spontaneous co-speech gestures may reveal:

• embodied and situated aspects of abstract reasoning

• materializations of spatial metaphor

• interaction of iconicity and metaphor (also in signed languages)

Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; McNeill 1992,

2005;

Müller 1998, 2003, 2008; Nuñez 2004; Nuñez & Sweetser 2006; Sweetser 1998, 2007;

Taub 2001; P. Wilcox 2000, 2004; S. Wilcox 2004; Williams 2008; inter alia

While metaphor is central to accessing abstract

domains

and the gestures in the data of this study are

essentially metaphorical in nature -

metaphorical modes do not suffice

to account for the semantic / pragmatic

processes in the multimodal data.

=> not necessarily a direct iconic link

between

the form of the gesture and

the form of its referent.

example:

palm-up open hand gesture on mention of “noun”

as sitting on palm-up open hand (Müller 2004)

contiguity relation between

hand and imaginary

object

Grammatical categories (noun, verb)contiguity relations between hands & imaginary objects

{video clip}

cognitive-semiotic approach to co-speech gesture

Charles Sanders Peirce’s (1931, 1955) pragmaticist theory of signs, focus on interpretationperspective for analysis: addressee

--------------Roman Jakobson’s (1956, 1960 1961,1963, 1966)

theory of metaphor & metonymy--------------Cognitive linguistics: metaphor, metonymy, image schemas

(Dirven 2002; Gibbs 1994, 2006; Johnson 1987; Lakoff 1987, 1993; Lakoff & Johnson 1980, 1999; Radden 2000; Panther & Thornburg 2003, 2004;

Sweetser 1990; Taub 2001; Wilcox 2004)

Gesture research (Calbris 1991; Cienki 1998; Kendon 2000, 2004; McNeill 1992, 2005; Müller 1998, 2004, 2007; Webb 1996; Streeck 2003; Sweetser 1998;

reseach project “Towards a grammar of gesture,“ Berlin / Viadrina FFO)

---------Data:

genre: academic discourse / introductory linguistics coursessubject matter: linguistic form, grammar, & linguistic theorygesture types: “referential gestures” of abstract concepts and structures (Müller 1998)

meta-linguistic & meta-grammatical co-speech gestures

inventory of gestural sign carriers in meta-linguistic & meta-grammatical gestures

data-driven approach resulted in a set of prominent hand configurations & motion patterns recurrent forms and recurrent form-meaning mappings

– manipulation of imaginary objects of different sizes and dimensions (solid objects and containers with an inside)

– geometrical shapes

basic shapes: squares, triangles, regtangles, (semi-) circles, etc.

lines: straight, curved, wavy along horizontal, vertical

und diagonal axes

• other patterns evoke some of image-schematic patterns suggested in the cognitive linguistics literature

=> ‘material’ basis for metaphorical and metonymic projections

Mittelberg (2006, 2008, fc)

Johnson‘s original definition of “image schemas”:

“recurring, dynamic patterns of our perceptual interactions and motor programs that give coherence and structure to our experience.”

(Johnson 1987, XIV)

Johnson (2005, 31):

“But let us not forget that the truly significant work done by image schemas is tied to the fact that they are not merely skeletons or abstractions. They are recurring patterns of organism-environment interactions that exist in the felt qualities of our experience, understanding and thought.”

image schemata instantiated in the gesture data

SUPPORT (“puoh-tray,” “puoh-cup”)CONTAINMENT (“puoh-cup,” “fist,” “pcoh-box”) OBJECT (“puoh-tray,” “puoh-cup,” “pcoh-box,”, “fist”)SOURCE-PATH-GOAL (“hori-trace,” “vert-trace,” “diag-trace; )BALANCE (“puoh-tray-bh,” “puoh-cup-bh,” “fist-bh,” “sym-offshoot”) SCALE (“scale”)CYCLE (“circle-bh,” “wrist-rotation,” “rotation lateral”)ITERATION (“wrist-rotation,” “rotation lateral”)FRONT-BACK (“push,” “pull”) FORCE (“push,” “pull”, “hori-join,” “sym-offshoot”) ADJACENCY CONTACT PART-WHOLE (Johnson 1987; Gibbs 2005; Hampe 2005; Lakoff & Johnson 1999; Mandler 1996; Talmy 1988)

Gesture research: iconic and metaphoric gestures (Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; Mittelberg 2008, fc.;

Müller 1998, 2007, 2008, fc.; Sweetser 1998; Williams 2004, 2008; Zlatev 2005; inter alia)

metonymic pathways

as emphasized by relevance theorists (e.g., Sperber & Wilson 2002) and cognitive linguists (Gibbs 1994; Lakoff 1987; Fauconnier & Turner 2002),

“metonymic pathways” are part of the cognitive competence of speakers and listeners

=> provide natural inference schemata (Panther & Thornburg 2003, 2004)

One definition of metonymy:

“Metonymy is a cognitive process in which one conceptual entity, the vehicle,

provides mental access to another conceptual entity, the target, within the same

domain or ICM.” (Kövecses and Radden 1998: 39)

“Proust is on the top shelf.”

Jakobson (1956): “Two types of language and two types of aphasic disturbances”

inspired by Peirce’s (1955) concepts of similarity & contiguity theory of metaphor & metonymy

as two major modes of association & signification

Jakobson

devotes equal attention to both tropes and those who use his work posit a continuum

between the two. (cf. Croft 1993; Dirven 2002; Lodge 1977; Radden 2000; Waugh

1998)

no absolute categories: they interact in any process of meaning-making to various degrees

hierarchy: the function of a given sign depends on the preponderance of one mode over the

others.

The aim here is to show how metonymic principles lead into the interpretation of

metaphoric gestures: metonymy first, metaphor second

Jakobson further distinguished between

internal metonymy:

contiguity relationships between sign & object synecdoche (pars pro toto)

“all hands on deck” (hands stand for entire bodies)

external metonymy:

contiguity relationships between signs or between entities in extra-linguistic, discursive, or conceptual space

“metonymy proper” (adjacency, contact)

“The White House remains silent.”

“The White House” stands for U.S. president / spokesperson (place for person)

I. internal metonymy (within metaphoric gestures)

the formation of the gestural sign

gestures exhibit the basic principle of semiotic representation

=> that is, that all semiotic representations are partial

these abstraction processes are driven by synecdoche:

by picking out the locally pragmatically salient aspects of an object / action

e.g., shape, dimensions, texture, manner of movementfor example, a gesture representing a picture frame

(Bouvet 2001; Cienki & Müller 2006; Gibbs 1994; Müller 1998, fc.; Sweetser 2001; Taub 2001; Wilcox 2004; Wilcox & Morford 2007)

most work on metonymy in co-speech gesture has focused on this kind of metonymic motivation

internal metonymy (within metaphoric gestures)

a horizontal line stands for a sentence“a sentence is a string of words”

external metonymy (within metaphoric gestures)

space extending between

fingers is

assigned meaning(adjacency/contact)

here:

the word “Diana”

in a sentence

external metonymy / metonymy of place

“sub-category” {video clip}

Conotiguity relations between body and objects

“there is”

pointing gesture

(“shifters”; Jakobson)

“the main verb”CONTAINMENT (cup)

metaphor

[object manipulation &

external metonymy]

“there is what‘s called the main verb”

Contiguity relations holding between hands & object

“sub-category”

imaginary OBJECT seemingly being held between two hands

[external metonymy]

comparably low position

in gesture space &

in relation to body

[metonymy of place]

metonymy first, metaphor second (Mittelberg & Waugh fc.)

1. Metonymy (external):

source: hands (physical)

target: object (imaginary)

2. Metaphor:

source: OBJECT (imaginary)

target: IDEA (conceptual)

co-speech: “sub-category“ (technical)

metonymic target = metaphoric source (imaginary object)

=> PLACE FOR FUNCTION

“sub-category”

some conclusions

The assumption that some metaphors are grounded in metonymy

seems to hold in the gestures discussed.

(Barcelona 2000; Dirven 2002; Fauconnier & Turner 2002; Geeraerts 2002; Goossens 1995; Jakobson 1956, 1960;

Lakoff 1987; Lodge 1977; Radden 2000; inter alia)

figures of thought may manifest themselves in gesture even if the concurrent speech is non-figurative

(“noun”, “Diane”, “teach-”, “infinitive”, “sentence”, etc.)

(cf. Cienki 1998, 2008; Cienki & Müller 2008,fc.; Müller 2008)

Gesture and cognitive linguistics

Alan Cienki

• Given:

– the goal in cognitive linguistics to characterize language in a way that is coherent with what is known about cognitive processing in general

– the cog.ling. focus on actual linguistic behavior– the universality of gesture with speech

Gesture can provide additional insight into conceptualizations underlying semantic and grammatical structures

Gesture in relation to the study of several topics in cognitive linguistics

• metaphor• cognitive models • mental spaces • metonymy and cognitive reference points• image schemas

Metaphor in words and gestures (Cienki 1998, 2008a)

• Gestures can provide evidence of activation (on some level) of an image being used to characterize a given topic

• (a mapping from one domain to another, i.e., [conceptual] metaphor, on some level)

• The inclusion of gesture data provides a way out of the criticism of circularity of conceptual metaphor theory

• (that is: language provides evidence of conceptual metaphors, and we find evidence of conceptual metaphors in language)

Metaphor in words and gestures (Cienki 1998, 2008a)

• Metaphor can appear in words without accompanying gestures

• Metaphor can appear in gestures without accompanying metaphorically used words

• The ‘same’ metaphor can appear in words and gestures in an utterance

• Different metaphors can be expressed simultaneously in words and gestures

Cognitive models

• A cognitive model as the way an individual characterizes knowledge for a given domain(compare schema, script, frame, etc.)

• Cultural models as shared cognitive models

• These models can be constituted by metaphors to varying degrees

Russian model of chestnost’

Dlia menia chestnost’ eto nekaia abso`liutnaia kategoriia

For me chestnost’ is a kind of absolute category.

Kogda vot iest’ situatsiia, When there’s this situation,

seichas postupit’ chestno `tak.then [you need] to act honestly like this.

{video clip}

Cognitive models and gesture (Cienki 1999)

• Metaphors can highlight aspects of cognitive/cultural models

• Gesture data can provide evidence of the speaker’s online use of a cognitive model

• This can be valuable for the study of (aspects of) cognitive models which are not expressed in words

Mental spaces

• Developed from work by Gilles Fauconnier (1985)

• Refers to “small conceptual ‘packets’ constructed as we think and talk”

• Linguistic forms – reveal the speaker’s use of various mental spaces, – and they provide cues to the listener for how to

construct the mental spaces used by the speaker

• Examples: hypothetical constructions (if…, then…)

Mental spaces: gestural example (Cienki 2008b)

It depends on thestudent,

but it also dependson the teacher.

{video clip}

Mental spaces, metaphor, and gesture

• IDEAS AS SPACES

(a kind of ontological metaphor, i.e., ABSTRACT AS CONCRETE)

Metonymy and reference points

• Langacker (1993) notes how we regularly make use of one (known) entity to invoke mental contact with another (less well-known) entity: – > cognitive reference point

• Metonymy as one means of doing this

Metonymy and reference points (Cienki 2007)

• Appears in gesture in many ways:– The hand enacts a function it would actually

perform (possibly with an object) (e.g., you have to write fast!)

– Or the hand represents (embodies) part of an object(viz. Darstellungsweisen [modes of representation], Müller 1998)

– Or the hand points to a reference point

Metonymy in a referential gesture

– hand points`That was dishonest,

’cause she didn’t…

• abstract targetgrounded onconcrete referent

{video clip}

Image schemas

“An image schema is a recurring, dynamic pattern of our perceptual interactions and motor programs that gives coherence and structure to our experience.” (Johnson 1987: xiv)

E.g.:CONTAINER BALANCEPATH COUNTERFORCECYCLE LINKPART-WHOLE CONTACTSURFACE CENTER-PERIPHERY

If we accept that...

• image schemas constitute significant patterns in our embodied experience

• most research on image schemas has been based on linguistic data

• manual gesture is a behavior involving a different kind of embodied experience than speech

• What role might image schemas play in our production, or comprehension, of gestures?

Image schemas and gesture (Cienki 2005)

• compared the categorization of two types of gestures in terms of image schemas:– Abstract Referential (metaphoric) gestures – combined group of discourse-structuring and

performative gestures (= “Other” gestures)

• two conditions: – gestures with no sound– gestures with speech

Image schemas and gesture (Cienki 2005)

• Can people reliably characterize the forms of gestures using image schemas as descriptors?

• Will this vary if they hear the accompanying speech or not?

Image schemas and gesture: findings (Cienki 2005)

• both groups of gestures were characterized with reliable agreement using a set of image schemas as descriptors – (this was true with and without sound)

• Abstract Referential (metaphoric) gestures were categorized with a slightly greater degree of agreement than the group of “Other” gestures (in both conditions)

Conclusions

• Cognitive linguistics claims to be a usage-based theory of language; gesture with speech is part of spoken language usage

• Taking this into account can lead to multimodal analyses of grammar (e.g., Fricke 2008)

Conclusions

• The study of gesture with speech prioritizes research on the dynamic processes of language and cognitive processing, including mental simulation

• Can help move cognitive linguistics beyond its focus on the individual speaker and connect it to research on interaction and intersubjectivity

Looking ahead

• Creativity in gesture (IM & AC)

• Open-class and closed-class systems in co-speech gesture (IM)

• Spoken language semantics (AC)

• Creativity in gesture (IM & AC)

Creativity in gesture: forms

– use of more extended gesture space– greater dynamicity– coordinated with other conspicuous

embodied expressions (e.g., body shifts, facial expression)

• Differences in scale between non-creative and creative gestures

Creativity in gesture: functions

– graphic representation of something concrete

– or graphic representation of an (abstract) idea via a physical image

–> metaphoric gesturing

– or clearly performing a comment reflecting the speaker’s attitude toward the topic

• Creative gestures appear to serve discrete functions (versus greater multifunctionality of less creative gestures)

Creativity in gesture:contexts of production

• Impetus to express a complex idea concisely

–> High communicative pressure– e.g., pedagogic context, or shared problem

solving• ongoing creative work• the “object” being presented needs to be

presented

• Open-class and closed-class systems in co-speech gesture (IM)

open-class & closed-class systems in co-speech gesture

In language, there is a fundamental difference between

content words (e.g., nouns, verbs, adjectives); open-class system (Talmy 2000)

function words (propositions, personal & demonstrative pronouns,

conjunctions, etc.) expressing spatial, temporal, logical and functional relations; closed-class system

How does the difference between substance and relations (structure) play out in co-speech gesture?

open-class & closed-class systems in co-speech gesture

two lines of inquiry:

A) Do gestures accompanying linguistic expressions of either kind differ in form

and function?

B) Is there a difference between gestures depicting contents and gestures

depicting relations - without considering the concurrent speech.

Since gestures are inherently indexical a focus will be on how they support or function as “shifters” (Jakobson 1960), e.g. those linguistic forms that take on a different meaning in a given moment of speech

• Spoken language semantics (AC)

Spoken language semantics

• Premises:– Inherent bias in formal linguistic research towards

language as it is written– Cognitive linguistic approach to semantics as

conceptualization– Different processes of conceptualization during

speaking versus writing, listening versus reading

• Thesis: The semantics of spoken language is different in form and content than that of written language

The semantics of spoken language as different in form and content than that of written language

• Spoken language functions differently than written language in terms of – time frame– grammatical structures used– embodied aspects of use– involvement of imagery– dynamic processes of expressing meaning

• Proposal: This calls for a distinct approach to analyzing the semantics of spoken language

Thank you!