Multimodal Interactional Analysis

Capturing Different Media

1. Sound2. Video3. Photo4. Observation (Field Notes)

MediaMedia ModesModesSoundSound Speech, music, laughter, background noiseSpeech, music, laughter, background noise

VideoVideo LightingLighting Ambient lighting in Ambient lighting in homes, daylight homes, daylight

Physical presencePhysical presence Posture, facial and Posture, facial and bodily gestures, eye bodily gestures, eye contact, pointingcontact, pointing

Production elementsProduction elements On screen effects, On screen effects, framing of shots, framing of shots, cutting between cutting between scenes, etc.scenes, etc.

Physical objectsPhysical objects Placing objects in field Placing objects in field of view, gesturing of view, gesturing with objects, with objects,

PhotographyPhotography Similar to video but without the Similar to video but without the movement/time aspectsmovement/time aspects

Observation Observation (field notes)(field notes)

Spoken language, physical behaviours, social Spoken language, physical behaviours, social dynamics, printed text/signs, etc. dynamics, printed text/signs, etc.

Multimodal Activity

Observe and record examples of how technologies influence the way that people relate to one another

Use one of the following media: 1.Sound Recording2.Observation (Field Notes)3.Still Photos4.Video Recording

Things to look out for?Reciprocity in communication

coordination of turn taking in conversation‘gift’ exchange

Ritualised behaviour openings and closings

Defining the ‘meaning’ of physical spaceWhat behaviours are

allowed/expected/normsReinforcement of these norms

Divided Consciousness/Absent PresenceDealing with immediate and distant social

contact simultaneously

Feedback SessionWhat modes have been captured?What different stories do they tell?Were there any issues with collecting/using

particular media?

1. Watch the videos/listen to recordings/view images

2. Identify themes◦ Observe and identify significant moments that highlight

multimodal interaction as an aspect of communication – note the time

◦ Refer to existing literature on multimodal aspects of communication to check what you see and hear – for example Goffman (1959)

3. Watch the videos4. Describe the multimodal aspects of the videos5. Roughly transcribe the video data to clarify significant

moments in communication and identify commonalities across videos

6. Watch the videos7. Final transcription in a suitable form to explain themes

and subthemes◦ Include standard annotation conventions

Developing Multimodal Themes

Multimodal transcription of video data (using my own coding scheme)

Multimodal Transcription

www.sigridnorris.com

Multimodal transcription – from conversational analysis:

(n) pause noted in seconds = joining of words : an extension of the preceding syllable; ; rising or falling intonation for subsequent

utterance underlined text indicates spoken with

particular vocal energy - a sudden cut off to an utterance; ? a rising tone . a falling tone , a shifting continuous tone bridging

utterances.

Multimodal Description

ReferencesClark, H. H. (1992). Arenas of language use. Chicago: The

University of Chicago Press.Clark, H. H., & Schaefer, E. F. (1989). Contributing to

discourse. Cognitive Science, 13, 259–294Kendon, A., Harris, R. M., & Ritchie Key, M. (1975).

Organization of behavior in faceto-face interaction. Paris: Mouton Publishers.

Kendon, A. (1990). Conducting interaction: Patterns of behavior in focused encounters.Cambridge: Cambridge University Press.

Goffman, E. (1963). Behavior in public places: Notes on the social organization of gatherings. New York: The Free Press.

Ten Have, P. (1999). Doing conversation analysis. A practical guide. London: Sage Publications.