A Window on Reality: Perceiving Edited Moving Images

7/31/2019 A Window on Reality: Perceiving Edited Moving Images

http://slidepdf.com/reader/full/a-window-on-reality-perceiving-edited-moving-images 1/22



3

We spend a fifth of our waking lives watching movies, television or some other

form of edited moving-image (Bureau of Labor Statistics, 2010). Nonetheless, until

recently research exploring the cognitive and perceptual bases of film has been a

lonely activity. Near the inception of film some astute psychological analyses

appeared (Arnheim, 1957; Münsterberg, 1916/1970; Pudovkin, 1929) but interest

waned, leaving only a handful of researchers (Carroll & Bever, 1976; Hochberg &

Brooks, 1978; Kraft, 1987) who, over the last 50 years, explored the profound

questions underlying our near universal ability to understand film. Part of the reason

for this isolation has been the presumption that the discontinuous stream of images

characteristic of film renders it utterly different from real-world perception, and only

understandable as a case of media-specific learning (Messaris, 1994). However,

recent film theory, a renewed interest in research exploring natural events, and the

recent availability of suitable technical and conceptual tools have conspired to end

this drought. In this paper, we review this research, and argue that it supports a

cognitive science of film viewing by making clear how film constitutes a form of

focused continuity that maintains consistency in cues that natural vision relies upon,

while allowing discontinuity in features and events that vision does not countenance.

Continuity Editing Reflects Natural Shifts of Attention Within Scenes.

In the 1890s most films depicted simple real-world scenes or staged narratives

filmed in a single run (a shot ) from a static camera. Shortly thereafter filmmakers

combined multiple shots – separated by cuts (sharp discontinuities from one shot to the

next), dissolves, fades, or wipes -- to create more compelling visual narratives. A suite of

staging, filming, and editing conventions emerged through trial and error that allowed

viewers to effortlessly integrate these diverse views. Most of these conventions, known as



4

continuity editing rules (or Hollywood style), were in common usage by 1918 (Bordwell,

Staiger, & Thompson, 1985; Bordwell & Thompson, 2001). Moreover, they permeate

much of visual media today.

Filmmakers believe that by adhering to continuity editing rules they can make a cut

“invisible” (Dmytryk, 1986) and ensure that “the spectator’s illusion of seeing a

continuous piece of action is not interrupted” (Reisz & Millar, 1953; pg. 216). For

example, a typical scene often begins with a shot of the location and the characters within

it (an establishing shot; Figure 1a) then progresses to medium shots framing the

characters involved in the action, close-ups of objects of interest (Figures 1c, 1d, 1e, and

1f), or shots of the speaker over the shoulder of the listener (Figure 1b). This gradual

honing in allows an unfolding of important details in a manner similar to how an observer

attends in the real world (Münsterberg, 1916/1970). This observation has been recently

confirmed by the application of high-speed infrared eyetracking to dynamic scenes

(Mital, Smith, Hill, & Henderson, 2011). Due to acuity limitations, a viewer must move

her eye (perform a saccadic eye movement ) if she wishes to see a part of a visual scene in

detail. This action projects the area of interest in the world onto the most sensitive part of

the retina, the fovea. After a saccade the eyes are relatively stable ( fixated ) and visual

encoding can occur. The sequence of saccades and fixations made during a dynamic

scene mirrors the honing into a scene typically seen within the Hollywood Style (Smith,

in press). Viewers attend to areas of high motion, faces of people engaged in conversation

(Mital, et al., 2011), and objects relevant to the viewing task (Smith & Henderson, 2008).



5

Figure 1: A stereotypical sequence of shots. a) Scene begins with an establishing shot

depicting both characters engaged in a conversation (character A, screen left;

character B, screen right); b) Medium over-the-shoulder shot (OTS) favoring

character A. Notice the color change of the scarf from shot 1; c) Close up (CU) of

character B as she glances out of shot; d) Cut away close-up of target of her gaze; e)

Close up of A; f) Close up of B from a camera positioned across the axis of action,

resulting in a reversal of screen direction, a violation of continuity.

The timing of these shifts is yoked to the timing of cuts, and vice versa. For

example, a sudden onset of motion - such as a character’s head turn or hand movement -

attracts viewer attention and masks the ability to detect disruptions (D. T. Levin &

Varakin, 2004). A film editor often uses the same motion to cut to a different view of the

scene (a match-action cut; Figure 2). This gives the impression of continuous flow and

minimizes viewer awareness of the cut. In a cut detection task, Smith and Henderson

(2008) demonstrated that a third of all match-action cuts were missed, whereas only a

tenth of cuts between scenes were missed (Figure 2). Continuity editing rules also use

other natural attention cues such as conversational turns, character gaze shifts, and

pointing gestures (Smith, in press). By piggy-backing on natural visual cognition,

Hollywood style presents a highly artificial sequence of viewpoints in a way that is easy



6

to comprehend, does not require specific cognitive skills, and may even be perceived by

viewers who have never watched film before (Schwan & Ildirar, 2010).

Figure 2: The top panel shows four types of cut from Blade Runner (Ridley Scott,

1982) that are between different scenes, between two viewpoints of same scene,

within a scene cued by an onset of motion, and within a scene cued by a glance out of

shot. The bottom panel shows the percentage of each type of cut missed during a cut

detection task. Adapted from Smith & Henderson (2008).

Statistical Regularities in Visual Features Within and Across Scenes

Contemporary films generally have shorter-duration shots, more motion, and are

darker than earlier films (see Figure 3; Bordwell, 2006; Cutting, Brunick, DeLong,



7

Iricinschi, & Candan, 2011; Salt, 2009). In addition, the fades, dissolves, and wipes that

were once common now bind only about 1% of all shots (Carey, 1974; Cutting, Brunick,

& DeLong, 2011). If we assume that cuts and motion force the reorientation of viewer

attention, and that darker images give viewers fewer options of where to look, then all of

these changes reflect filmmakers’ increasing control over viewers’ attention.

Figure 3: Representations of change in a sample of 160 Hollywood-style films

(dramas, comedies, animations, action films, and adventure films) released from 1935

to 2010. The left panel shows how average shot durations have declined from about

15 to about 3.5 s over 75 years. The middle panel uses an index of motion across all

frames of each film (averaging ~165,000/film) and shows how motion in films has

increased with time, and the right panel uses an index of brightness for all pixels in all

frames of each film and shows how films have gotten darker with time. Adapted from

Cutting, Brunick, DeLong, Iricinschi, and Candan (2011).

Nonetheless, films are not homogeneous in shot duration, motion, luminance, or in

any other dimension. Different scenes tend to have different rhythms across all of these

dimensions (Bordwell & Thompson, 2001; Murch, 2001). Systematic patterns underlie

this variability. For example, an establishing shot tends to be longer than the shots that

follow. This extra time may help viewers reorient their attention after the break in

continuity from the scene before. In addition, the final shot of a scene is often as long as

the establishing shot (Cutting, Brunick, & Candan, submitted). This extra time may be

used to focus attention on the dangling cause, the information in plot development that



8

drives the motivation for a change to the next scene (Bordwell, 2006; Thompson, 1999).

Thus, scenes really do have beginnings, middles, and ends, and the shots of the beginning

and end are often longer.

Whole films are characterized by rhythmic fluctuations that appear to guide viewer

attention. Gilden (2001) and others have suggested that human attention, as it is

distributed over tens of minutes and recorded in the form of reaction times, can reveal a

1/ f pattern ( “one over f”). Fourier analysis of a reaction time series decomposes it into

many sine waves of different frequency and power (proportional to the square of the

amplitude, or “height,” of the wave). A 1/ f pattern is a fractal (self-similar) pattern

distinguished by all the component sine waves having a power that is proportional to their

wavelength (or 1/ f requency). Interestingly, our minds also generate a 1/ f pattern of

attention. Since about 1960 the shot lengths of films, taken as a series, have increasingly

approached a 1/ f pattern as well (Cutting, DeLong, & Nothelfer, 2010). Given that films

occupy our minds and drive attention, it seems fitting that the shot-duration patterns of

popular film might increasingly be like those that our minds naturally generate.

The Common Experience of Visual Narrative

Driving the attention of a single viewer is insufficient; mainstream film must work

for all. When multiple viewers attend to the same static scene they do not necessarily

look in the same places at the same time (Mannan, Ruddock, & Wooding, 1997). It might

be expected that an increase in complexity of a dynamic scene would also increase

differences among viewers. However, the opposite is true. The gaze of viewers is highly

coordinated in dynamic scenes, a phenomenon called attentional synchrony (see Figure 4;

Smith, 2006). This synchrony is manifest as the spontaneous clustering of the gaze



9

positions across multiple viewers when free-viewing dynamic naturalistic scenes or

edited sequences (Mital, et al., 2011).

Figure 4: The gaze locations of 19 viewers while watching a short dialog sequence.

Viewers either saw the sequence with audio (yellow spots) or without (pink spots).

Top row: a cut cued by a shift in dialogue from the woman to the man. Bottom row: a

match-action cut cued by the woman’s head turn. Frame numbers are printed below

each frame. The tight clustering of gaze on faces following cuts (~360-400ms post-

cut; 9-10 frames at 25 frames per second) suggests an ease of reorienting to the new

content and a coordination across viewers known as attentional synchrony. Reprinted

with permission from Smith (2006).

Synchrony of viewer experience was hypothesized by the film editor, Walter

Murch (2001). He noticed that when editing The Conversation (Francis Ford Coppola,

1974) he often felt compelled to blink when Gene Hackman's character blinked on screen

and also to select that moment to cut. He hypothesized that the synchronization of blinks

between the entire audience and the actors would entrain viewer thoughts to the narrative.

Blink synchrony between viewers during film viewing was recently confirmed by

Nakano and colleagues (Nakano, Yamamoto, Kitajo, Takahashi, & Kitazaw, 2009). They



10

showed that viewers delayed blinks until the end of events and mirrored blinks of a

speaking actor with a delay of 250-500 milliseconds (Nakano & Kitazawa, 2010). As

Murch (2001) suggested, blinks may indicate a perceptual break in communication that

an editor can use to hide a cut. However, explicit measurement of the co-occurrence of

viewer eye blinks with cuts during a cut detection task indicated that most cuts did not

coincide with blinks although a significant number of within-scene cuts that were missed

did (11.6%; Smith & Henderson, 2008). Blink synchronization may be evidence of

attentional synchrony across viewers and entrainment to certain features including

dialogue and narrative events.

This synchronization of viewer experience has also been found in functional

Magnetic Resonance Imaging (fMRI). Hasson and colleagues (2004) showed the opening

30 min of The Good, the Bad and the Ugly (Leone, 1966) to participants in a scanner.

Inter-observer comparisons of brain responses during free viewing of the movie revealed

that activity in 45% of the neocortex was highly similar across viewers. These areas

included regions involved in vision, hearing, language processing, emotion, and multi-

sensory integration. This neural similarity was also shown to increase with the degree of

attentional synchrony (Hasson et al., 2008).

A similar synchrony has been observed in how viewers perceive event structure

depicted in film and real-world dynamic scenes. If multiple viewers are shown a video of

a simple activity, such as folding laundry, and are asked to press a button when one

meaningful event ends and another begins there will be considerable agreement

(Newtson, 1973). Event segmentation is believed to be critical for the efficient

distribution of cognitive resources, optimizing the organization in space and time of key

features in memory (Zacks, 2010).



11

Spontaneous event segmentation has also been shown to occur while watching

edited film sequences (Zacks, Speer, Swallow, & Maley, 2010). Viewers free-viewed the

classic children's film, The Red Balloon (Larmorisse, 1956) while their brain activity was

recorded. Afterwards, viewers segmented the film into events. Cuts that coincided with

large changes in action were both identified as event boundaries and induced brain

activity in areas of the cortex associated with motion processing (the human MT) and

attentional control (dorsal frontal areas). By comparison, cuts that simply changed

viewpoint within a scene while maintaining continuity of time, location, and action were

not identified as boundaries either behaviorally or by brain activity. Instead, such cuts

produced activity in cortical areas associated with attentional control and task switching,

possibly confirming the role of attentional shifts in obscuring cuts (Smith, in press).

These studies suggest that the perception of dynamic scenes entails formulating

minimal expectations about event sequences and distributing attention appropriately. As

long as these expectations are satisfied continuity will be perceived. This focused

continuity prioritizes only the audiovisual features of a scene that are important to the

viewer at the moment, allowing irrelevant features such as background details to change

without being noticed or violating a priori assumptions of continuity (Smith, in press).

During film viewing these expectations are preserved by editing conventions, allowing

the viewer to orient quickly to new shots and to check whether their expectations are

satisfied. Future investigations of event perception during film viewing should focus on

uncovering the extent and form of these expectations. These may in turn reveal how

malleable such expectations are during real-world event perception.

The Representational Basis of Film

Editing an action at times when attentional shifts would naturally occur helps

ensure that film viewing is a lot like viewing the real world. But how is it possible that



12

the sudden change in viewpoint does not lead to confusion? A cut from a medium-shot to

a close-up (Figures 1b and 1c) creates visual transients, a sudden change in viewpoint,

enlargement and rotation of characters, and omission of many scene details. Such a

transition in a real-world scene would entail eye, head, and body movement over an

extended period of time, all of which provide feedback on the extent of the movement

enabling the viewer to update their spatial representations of the scene (Tatler & Land,

2011).

One answer to this question arises naturally from the focused continuity hypothesis,

which not only emphasizes how film is in some ways perceptually consistent, but also

makes clear that other forms of consistency are not particularly important. Thus, we may

lack expectations about object location and appearance that might be violated by the

change in viewpoint. Circumstantial evidence in support of this hypothesis comes from

continuity errors. Often striking changes in object appearance and placement across cuts

go unnoticed by viewers. Levin and Simons (1997) found that most viewers failed to

report a change in identity of an actor across a match-action cut or a change in costume or

props across a shot/reverse-shot sequence during a dialog (similar to the change in scarf

color between shots 1 and 2 in Figure 1). However, change blindness was not absolute;

33% of viewers spontaneously reported seeing such changes, and the rate of detection

increased when the object was of greater interest. These results suggest that we don’t

represent all details of a scene; instead we represent only enough to follow the narrative.

But what form do these representations take? If we were to form a coherent 3D,

allocentric (map-like) representation of the depicted scene, it would entail mentally

rotating each camera viewpoint and matching details across shots. This would seem to

require significant mental effort. Recent empirical investigations of space and object

memory across cuts suggest that we represent edited sequences differently than we



13

represent the real world. During real-world tasks involving navigation of the environment

we often shift our attention to objects that were previously fixated but are currently

beyond the visual field, such as saccading to a kettle that is behind us when making tea

(Tatler & Land, 2011). Thus, we appear to retain at least rudimentary allocentric

representations of task-relevant object locations. These locations allow us to maintain

temporarily limited information concerning appearance and identity.

Across a cut, memory for object location in a scene is significantly worse than

memory for object identity, or color (Hirose, Kennedy, & Tatler, 2010). Memory for

relations among locations is also poor even after many hours of exposure to the same

set (D. Levin, 2010). Only when a scene is repeatedly presented from the same

viewpoint, as occurs for sitcoms filmed in front of studio audiences, are viewers able

to accurately remember the spatial relationship between locations (Levin, 2010;

Figure 5). This evidence suggests that an allocentric representation of scenes is not

generally retained during film viewing. Future research can determine whether this is

due to a failure to represent spatial information or to the use of a different reference

frame (e.g. egocentric, viewpoint-dependent) and how featural information within it is

associated with these unreliable locations.



14

Figure 5: Participants were shown a series of two stills from popular TV shows (top

Friends, bottom ER) and asked to judge whether the second location was to the right

or left of the first. Mean percentage correct is displayed as a function of whether the

show had constrained views (e.g. limited camera positions due to being shot in front

of a studio audience) or multiple views (e.g. presented via a mobile camera) and how

many times the participant had viewed each show ( Number of Viewings). Adapted

from Levin (2010). Performance improved with increasing exposure to constrained-

view shows but not for multiple-view shows suggesting that viewers only retain

limited spatial knowledge of scene layout.



15

Summary

In this article we have reviewed empirical investigations in film perception and

the cognitive foundations of the dominant conventions of film form known as the

continuity editing rules. Film has existed for over a century and early psychologists

wondered how we perceive edited moving images (Münsterberg, 1916/1970). However,

it is only with the recent advent of new methodologies in cognitive psychology -- such as

eye tracking, and fMRI – that we have a new willingness to deal with more naturalistic

stimuli in the study of visual cognition. Thus, researchers are beginning to understand the

cognitive processes involved in that universal pastime -- watching movies.



16

Acknowledgments

We would like to thank Jeff Zacks and two anonymous reviews for their constructive

comments on an earlier draft of this manuscript. Elements of this review were

presented in the "Psychocinematics" symposium organized by Art Shimamura at the

52nd Annual Psychonomics Society annual meeting, Seattle, November 2011 and at

past meetings of the Society for Cognitive Studies of the Moving-Images (SCSMI;

http://scsmi-online.org/). Tim J. Smith was partly funded by Leverhulme Trust

(Ref F/00-158/BZ) on the Dynamic Images and Eye Movement project (DIEM:

http://thediemproject.wordpress.com/ ).

http://scsmi-online.org/


http://thediemproject.wordpress.com/






17

Recommended Readings

Anderson, J. (1996). The Reality of Illusion: An Ecological Approach to Cognitive Film

Theory. Carbondale: Southern Illinois University Press. A film maker with a

background in psychology tries to understand moviemaking with an eye towards

our biological make-up.

Levin, D. & Wang, C. (2009). Spatial representations in film, Projections. A review of

space perception across edited moving images.

Smith, T. J. (in press) The attentional theory of cinematic continuity, Projections. A

summary of how we attend to dynamic scenes and a cognitive theory of continuity

editing.

Zacks, J. M., & Magliano, J. P. (2011). Film understanding and cognitive neuroscience.

In D. P. Melcher & F. Bacci (Eds.), Art & The Senses (pp. 435-454). New York:

Oxford University Press. A comprehensive overview of the relevance of event

perception to perceiving film narratives.



18

Filmography

Ford Coppola, F. (1974). The Conversation. USA

Larmorisse, A. (1958). The Red Balloon. France.

Leone, S. (1966). The Good, the Bad and the Ugly. Italy/Spain/West Germany.

Scott, R. (1982). Blade Runner. USA



19

References (40)

Arnheim, R. (1957). Film as Art . Berkeley, California: University of California Press.

Bordwell, D. (2006). The Way Hollywood Tells It: Story and Style in Modern Movies. Los

Angeles, US: University California Press.

Bordwell, D., Staiger, J., & Thompson, K. (1985). The Classical Hollywood Cinema: Flm Style &

Mode of Production to 1960. London, UK: Routledge.

Bordwell, D., & Thompson, K. (2001). Film Art: An Introduction (Vol. 6th). New York, USA: Mc

Graw Hill.

Bureau of Labor Statistics. (2010). American Time Use Survey.

Carey, J. (1974). Temporal and spatial transitions in American fiction films. Studies in the

Anthropology of Visual Communication, 1, 45-50.

Carroll, J. M., & Bever, T. G. (1976). Segmentation in Cinema Perception. Science, 191(4231),

1053-1055.

Cutting, J. E., Brunick, K. L., & Candan, A. (submitted). Perceiving event dynamics and parsing

Hollywood films. .

Cutting, J. E., Brunick, K. L., & DeLong, J. E. (2011). The changing poetics of the dissolve in

Hollywood film. Empirical Studies in the Arts, 26, 149-169.

Cutting, J. E., Brunick, K. L., DeLong, J. E., Iricinschi, C., & Candan, A. (2011). Quicker, faster,

darker: Changes in Hollywood film over 75 years. i-Perception, 2, 569-576.

Cutting, J. E., DeLong, J. E., & Nothelfer, C. E. (2010). Attention and the evolution of

Hollywood film. Psychological Science , 21, 440-447.

Dmytryk, E. (1986). On Filmmaking. London, UK: Focal Press.

Gilden, D. L. (2001). Cognitive emission of 1/f noise. Psychological Review (108), 33-56.

Hasson, U., Landesman, O., Knappmeyer, B., Valines, I., Rubin, N., & Heeger, D. J. (2008).

Neurocinematics: The Neuroscience of Film. Projections: The Journal of Movies and

Mind, 2(1), 1-26.



20

Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., & Malach, R. (2004). Intersubject synchronization

of cortical activity during natural vision. Science 303, 1634-1640.

Hirose, Y., Kennedy, A., & Tatler, B. W. (2010). Perception and memory across viewpoint

changes in moving images. Journal of Vision, 10(4), 1-19.

Hochberg, J., & Brooks, V. (1978). The Perception of Motion Pictures. In E. C. Carterette & M.

P. Friedman (Eds.), Handbook of Perception (Vol. X). New York, US: Academic Press.

Kraft, R. N. (1987). Rules and strategies of visual narratives. Perceptual and Motor Skills, 64,

3-14.

Levin, D. (2010). Spatial representation of the sets of familiar and unfamiliar television

programs. Media Psychology, 13, 1-23.

Levin, D., & Simons, D. J. (1997). Failure to detect changes to attended objects in motion

pictures. Psychonomic Bulletin and Review (4), 501-506.

Levin, D. T., & Varakin, D. A. (2004). No pause for a brief disruption: Failures of visual

awareness during ongoing events. Consciousness and Cognition, 13, 363-372.

Mannan, S. K., Ruddock, K. H., & Wooding, D. S. (1997). Fixation sequences made during

visual examination of briefly presented 2D images. Spatial Vision, 11, 157-178.

Messaris, P. (1994). Visual Literacy: Image, Mind, and Reality . Boulder, USA: Westview Press.

Mital, P. K., Smith, T. J., Hill, R. L., & Henderson, J. M. (2011). Clustering of gaze during

dynamic scene viewing is predicted by motion. Cognitive Computation, 3(1), 5-24.

Münsterberg, H. (1916/1970). The film: a psychological study; The silent photoplay in 1916.

New York, USA: D. Appleton and Company.

Murch, W. (2001). In The Blink Of An Eye: a perspective on film editing. Los Angeles, USA:

Silman-James Press.

Nakano, T., & Kitazawa, S. (2010). Eyeblink entrainment at breakpoints of speech.

Experimental Brain Research, 205(4), 577-581.



21

Nakano, T., Yamamoto, Y., Kitajo, K., Takahashi, T., & Kitazaw, S. (2009). Synchronization of

spontaneous eyeblinks while viewing video stories. Proceedings of The Royal Society

B, 276, 3635-3644.

Newtson, D. (1973). Attribution and the unit of perception of ongoing behavior. Journal of

Personality and Social Psychology (28), 28-38.

Pudovkin, V. (1929). On Film Technique. London: Gollancz.

Reisz, K., & Millar, G. (1953). Technique of Film Editing. London, UK: Focal Press.

Salt, B. (2009). Film Style and Technology: History and Analysis (Vol. 3rd). Totton, Hampshire,

UK: Starword.

Schwan, S., & Ildirar, S. (2010). Watching film for the first time: How adult viewers interpret

perceptual disconuities in film. Psychological Science, 21(7), 1-7.

Smith, T. J. (2006). An Attentional Theory of Continuity Editing. Ph.D., University of

Edinburgh, Edinburgh, UK.

Smith, T. J. (in press). Attentional Theory of Cinematic Continuity. Projections.

Smith, T. J., & Henderson, J. M. (2008). Edit Blindness: The Relationship Between Attention

and Global Change Blindness in Dynamic Scenes. Journal of Eye Movement Research,

2(2), 1-17.

Tatler, B. W., & Land, M. F. (2011). Vision and the representation of the surroundings in

spatial memory. Philosophical Transactions of the Royal Society B, 366(596-610).

Thompson, K. (1999). Storytelling in the New Hollywood . Cambridge, MA: Harvard University

Press.

Zacks, J. M. (2010). How we organize our experience into events. Psychological Science

Agenda, 24.

Zacks, J. M., Speer, N. K., Swallow, K. M., & Maley, C. J. (2010). The brain’s cutting-room

floor: segmentation of narrative cinema. Frontiers in Human Neuroscience, 4.

A Window on Reality: Perceiving Edited Moving Images

Documents

Transcript of A Window on Reality: Perceiving Edited Moving Images