Emotional Eyes: from Sensory Evolution to Social

Exaptation

by

Daniel Hyuk-Joon Lee

A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Psychology

University of Toronto

© Copyright by Daniel H. Lee 2015

ii

Emotional Eyes: from Sensory Evolution to Social Exaptation

Daniel H. Lee

Doctor of Philosophy

Department of Psychology University of Toronto

2015

Abstract

“The eyes are the windows to the soul.” Cicero’s quote captures how readily we use the eyes of

facial expressions to communicate our internal states. However, we do not yet know how this

ability came about. Darwin theorized that our emotional expressions evolved as functional

adaptations for the expresser, which were then co-opted as social signals. Here, I set out to

empirically bridge this gap in our understanding by showing how our facials expressions

originated for sensory egocentric function, and how those origins were socially co-opted to serve

expressions’ contemporary allocentric function as signals of mental states. To do this, I focused

on the eyes and their surrounding features, adopting Darwin’s principles that the expressive

features associated with the eyes’ widening versus narrowing can be understood as opposing

along a continuum of sensory regulation. For the expresser, I show how eye widening in fear

versus eye narrowing in disgust expressions serve opposing sensory functions. Eye widening

increased the visual field and light gathering to enhance visual sensitivity versus eye narrowing

which increased light focusing to enhance visual acuity, adaptively serving fear’s function in

vigilance toward threat detection versus disgust’s function in threat discrimination, respectively.

For the expression’s observer, eye widening enhanced the physical gaze signal to better transmit

the direction of a mutual threat detected by fear. Further, the widening versus narrowing eye

features that enhanced perceptual sensitivity versus discrimination similarly conveyed opposing

iii

mental states of information sensitivity versus discrimination. Together, the evidence connects

the appearance of our expressions from their egocentric origins to their modern-day allocentric

functions. Thus by the same eyes with which we see the world, we are provided windows not just

into another’s soul, but to a co-evolved history of how our individual survival was leveraged into

a flourishing co-operative one.

iv

Acknowledgments First and foremost, I thank my academic supervisor and mentor, Adam Anderson, from whom I

learned the invaluable art of science as much as the valued science itself. I thank my thesis

supervisors, Susanne Ferber and Nick Rule, for their guidance in this and further research

beyond. I thank the many research assistants, Jennifer Chu, Andrew Dakin, Zoe Katsiapis,

Whiwon Lee, Rameez Mahmood, Andrew Micieli, and Reza Mirza, whose time and dedication

made this research possible. I thank the members of the Affect and Cognition Lab: the post-

doctoral fellows Rebecca Todd, Assaf Kron, and Junichi Chikazoe, who showed me how this

journey was possible, and my fellow graduate students, Josh Susskind, Norman Farb, Hanah

Chapman, and Taylor Schmitz, who not only took the first steps but also looked back so that I

could, in confidence, follow. Lastly I thank my family, my mother and father, towards whom my

gratitude cannot be expressed in words.

v

Table of Contents ACKNOWLEDGMENTS ........................................................................................................................................ IV

TABLE OF CONTENTS ........................................................................................................................................... V

LIST OF FIGURES ................................................................................................................................................. VII

LIST OF APPENDICES ....................................................................................................................................... VIII

1 INTRODUCTION .............................................................................................................................................. 1

1.1 FACIAL EXPRESSIONS: EMPIRICAL ORIGINS ..................................................................................................... 2 1.1.1 Categorical View ................................................................................................................................... 3 1.1.2 Opposition to the Categorical View ....................................................................................................... 4

1.2 FACIAL EXPRESSIONS: THEORETICAL ORIGINS ................................................................................................ 6 1.3 THESIS: SENSORY EVOLUTION TO SOCIAL EXAPTATION .................................................................................. 8

1.3.1 The Eyes ................................................................................................................................................. 9 1.3.2 Outline of Studies ................................................................................................................................. 10

2 EGOCENTRIC SENSORY BENEFITS OF EXPRESSIVE EYES ............................................................ 11

2.1 GENERAL METHODS AND MATERIALS ........................................................................................................... 12 2.1.1 Participants ......................................................................................................................................... 12 2.1.2 Posing Facial Expressions and Validation .......................................................................................... 12

2.2 EXPERIMENT 1: EGOCENTRIC VISUAL FIELD .................................................................................................. 13 2.3 CHAPTER DISCUSSION ................................................................................................................................... 17

3 SOCIAL SENSORY BENEFITS OF EXPRESSIVE EYES ........................................................................ 20

3.1 GENERAL METHODS: EYE MODELING ........................................................................................................... 21 3.2 EXPERIMENT 2A: GAZE DIRECTION ENHANCEMENT ...................................................................................... 22 3.3 EXPERIMENT 2B: PHYSICAL VERSUS EMOTIONAL EYE SIGNALS ..................................................................... 25 3.4 EXPERIMENT 3: PERIPHERAL TARGET LOCALIZATION .................................................................................... 27 3.5 CHAPTER DISCUSSION ................................................................................................................................... 31

4 OPTICAL ORIGINS OF EXPRESSIVE EYE WIDENING ....................................................................... 34

4.1 OPTICAL MODEL ............................................................................................................................................ 35 4.1.1 Sensitivity ............................................................................................................................................. 36 4.1.2 Acuity ................................................................................................................................................... 36

4.2 EXPERIMENT 4: MEASURING SENSITIVITY ..................................................................................................... 38 4.3 EXPERIMENT 5: MEASURING ACUITY ............................................................................................................ 40 4.4 CHAPTER DISCUSSION ................................................................................................................................... 42

5 READING WHAT THE MIND THINKS FROM HOW THE EYE SEES ................................................ 45

vi

5.1 GENERAL METHODS: EXPRESSION MODELING .............................................................................................. 46 5.2 EXPERIMENT 6: MENTAL STATES MAP ........................................................................................................... 47 5.3 EXPERIMENT 7: READING EYES IN MIXED EXPRESSIONS ................................................................................. 49 5.4 CHAPTER DISCUSSION ................................................................................................................................... 52

6 SUMMARY AND CONCLUSIONS .............................................................................................................. 55

REFERENCES .......................................................................................................................................................... 60

APPENDIX A: STIMULUS SETS ........................................................................................................................... 67

vii

List of Figures FIGURE 2.1: EGOCENTRIC VISUAL FIELD CHANGES WITH EXPRESSION ................................................................... 16

FIGURE 3.1: GAZE DIRECTION ENHANCEMENT WITH WIDER EYE OPENING ............................................................. 24

FIGURE 3.2: COMPARED EFFECTS OF UPRIGHT AND INVERTED EYES ....................................................................... 27

FIGURE 3.3: REVEALED IRIS AREA PREDICTS RESPONSE TIME TO CORRECTLY CUED PERIPHERAL TARGETS ....... 30

FIGURE 4.1: MODELED OPTICAL EFFECTS OF EYE APERTURE .................................................................................. 38

FIGURE 4.2: MEASURED PERCEPTUAL EFFECTS OF EYE APERTURE ......................................................................... 40

FIGURE 5.1: RELATIONSHIP BETWEEN MENTAL STATES BASED ON EYE FEATURES ................................................. 49

FIGURE 5.2: EFFECT OF EYES ON MENTAL STATE PERCEPTION IN FULL AND MIXED EXPRESSIONS ........................ 52

viii

List of Appendices APPENDIX A: STIMULUS SETS ........................................................................................................................... 67

1

1 Introduction Facial expressions of emotion are a salient and important means of social communication.

Visually, the face is the most important part of an individual’s identity. So it is not surprising that

how our facial information changes by its expressivity serves as an important source of

nonverbal communication (Bruce & Young, 1986; Calder & Young, 2005) and captures our

attention (Vuilleumier, Armony, Driver, & Dolan, 2001). The importance of facial expressions is

also reflected in psychological research. As visible and outward signals of our otherwise hidden

internal states, facial expressions played a prominent role in the early stages of contemporary

emotion research (Ekman, Sorenson, & Friesen, 1969), and it still remains a rich topic of

research today, as indicated by its volume of work1. Because of this work, we have accrued

much understanding of what mental states our facial expressions communicate (e.g., fear,

disgust, happy, sad; Etcoff & Magee, 1992; Jack, Garrod, Yu, Caldara, & Schyns, 2012; Young

et al., 1997).

However, in spite of the empirical progress that has been made in what our facial expressions

communicate, there remains a critical gap in our understanding: we do not know why or how

they came about. While our contemporary use of facial expressions are predominantly used for

communication, we do not know whether these expressive actions are socially learned

behaviours for communication (Fridlund, 1997) or were evolutionarily selected for some

adaptive purpose for the expresser (Darwin, 1872). For example, although we can recognize a

fear expression with high accuracy (Susskind, Littlewort, Bartlett, Movellan, & Anderson, 2007)

and point to the specific facial features that make it recognizable as “fear” (Ekman, Friesen, &

Hager, 1978; Susskind et al., 2008), we do not know why those features (e.g., eye widening,

brow raising) signal “fear”. Are these communicative features a product of social learning, such

that any set of facial features could have been substituted? Or were the features of fear

specifically selected for some evolutionary purpose, and if so, for what purpose?

In my dissertation, I set out to bridge in this gap in our understanding from the evolutionary

perspective of our facial expression appearance (Darwin, 1872). I argue that our facial features

1At time of writing, a Pubmed search on “facial expression” and “emotion” returned 5360 articles.

2

originated for an immediate egocentric function for survival. Then, I show how these same

features that are grounded in adaptive function underwent social exaptation for their

contemporary purposes of communication.

In order to empirically test this evolutionary perspective of our facial expressions’ origins, I

focus on the eyes and the surrounding features that alter their appearance. This encompasses

several reasons. The importance of vision for our evolutionary survival is highlighted by the

disproportionate volume of cortex dedicated to visual processing, compared to our other senses.

Meanwhile, the focused examination of just the eye features makes tractable the testing of

specific hypotheses within the human facial expression space, which is combinatorially vast,

supported by a large number of independent features (Ekman, Friesen, & Hager, 1978). Last but

not least, the eyes are an important source of social signaling with the capacity to communicate a

wide variety of complex mental states (Baron-Cohen, Wheelwright, Hill, Raste, & Plumb, 2001).

The eyes are thus ripe for testing the question of how our expressive eyes’ sensory evolution may

have been socially co-opted for communication. But before delving into the studies that address

this question, let us first consider its empirical and theoretical background.

1.1 Facial Expressions: Empirical Origins We begin at the origins of modern facial expression research, claimed by Paul Ekman and

colleagues. Their two most significant contributions to our understanding of facial expressions

are: 1) discovering categories of facial expressions recognized consistently across cultures; and

2) taxonomizing the facial muscles that support the full repertoire of human facial expressions.

In the main empirical work supporting their first contribution, Ekman and colleagues tested

populations of literate and preliterate cultures (including United States, Japan, New Guinea,

Borneo) and found evidence for consistent recognition of six emotion categories (fear, disgust,

surprise, anger, happy, and sad; Ekman, Sorenson, & Friesen, 1969; Ekman & Friesen, 1971).

These discrete categories of facial expressions were called “basic expressions” (Ekman, 1999,

Izard, 1994) and considered to be universal (Ekman, Sorenson, & Friesen, 1969; Ekman, 1999),

as the simplest explanation was that a common evolutionary origin passed them down across all

humans, rather than learned independently within each culture.

3

The second contribution of Ekman and colleagues is the Facial Action Coding System (FACS;

Ekman, Friesen, & Hager, 1978; updated in 2002), a taxonomy of the facial muscle groups

(“action units”) whose combinatorial contractions constitute the full possible facial expression

space. In defining over 30 action units (some are grouped combinations of muscle groups),

visually identifiable at 5 different degrees of contraction, FACS allowed for the anatomical

classification and measurement of any human facial expression.

For historical context, it is important to note that these contributions by Ekman and colleagues

addressed a broader issue in the field of emotions, under which the study of facial expressions

was included. The scientific study of emotions had been hindered by the difficulty of measuring

our internal, subjective experiences, and the contributions of Ekman and colleagues were a

response to this challenge. That is, if these categories of basic expressions were indeed universal,

researchers could treat them as overt readouts of our covert emotional states, which could be

measured and analyzed using FACS, providing an objective solution to the problem of

subjectivity.

1.1.1 Categorical View

The basic expressions (or “categorical view”) of facial expressions became widely adopted in

expression research as supporting evidence accumulated. This included behavioural findings that

directly tested the identification of basic expressions (Etcoff & Magee, 1992; Fujimura, Matsuda,

Katahira, Okada, & Okanoya, 2012; Young et al., 1997), as well as their patterned recognition in

humans (e.g., Adolphs, Tranel, Damasio, & Damasio, 1994; Smith & Schyns, 2009) which are

similar to machines’ expression recognition (Susskind et al., 2007). Culturally yet uninfluenced

newborns exposed to different tastes have also been found to express some basic expressions

(Steiner, 1973; Rosenstein & Oster, 1988)—in particular disgust, with its origins theorized to be

grounded in our earliest evolutionary function of distaste (Chapman & Anderson, 2012;

Chapman, Kim, Susskind, & Anderson, 2009).

Another source of evidence supporting basic expressions came from a number of neurological

findings that showed relatively confined brain regions supporting their recognition, the lesions of

which impair specific expression perception, such as the amygdala for fear (Adolphs et al., 2005;

Adolphs, Tranel, Damasio, & Damasio, 1994), insula and the striatum for disgust (Calder,

Keane, Manes, Antoun, & Young, 2000; Phillips et al., 1997; Sprengelmeyer et al., 1996), and

4

ventral striatum for anger (Calder et al., 2004). Direct perception of basic expressions is also

associated with the neural activities of similar regions, such as the amygdala for fear expressions

(Anderson, Christoff, Panitz, De Rosa, & Gabrieli, 2003; Breiter et al., 1996; Morris et al.,

1996), insula and amygdala for disgust (Anderson et al., 2003), and amygdala for anger (Adams,

Gordon, Baird, Ambady, & Kleck, 2003; Morris, Öhman, & Dolan, 1998; ventral amygdala:

Whalen et al., 2001). Furthermore, some of these expressions (in particular fear) have shown to

be distinctly processed beneath our threshold of consciousness (Whalen et al., 2004; Whalen et

al., 1998) or when not attended to (Anderson et al., 2003; Morris, DeGelder, Weiskrantz, &

Dolan, 2001; Vuilleumier, Armony, Driver, & Dolan, 2001), suggesting a primitive functional

origin that supports such neural wiring.

With the accumulation of evidence, the categorical view became normative not just in facial

expression research but also extended into its parent field of emotions. That is, these expression

categories were used to infer similar categories of internal emotional states that reflected

discriminable constellations of bodily changes (Ekman, Levenson, & Friesen, 1983; Levenson

Ekman, Heider, & Friesen, 1992; Scherer & Wallbott, 1994; also see James, 1884) and patterns

of brain activation (Damasio et al., 2000; Phan, Wager, Taylor, & Liberzon, 2002). This supplied

ammunition to a central debate among emotion researchers of how our emotions are represented.

1.1.2 Opposition to the Categorical View

In the field of emotions, the main opposition to the categorical view is the constructivist or

dimensional view, which holds that emotions occupy a continuous space of states rather than

discrete categories. Within this continuous space, constructivists emphasize the role of culture

and context that impose the categorical labels rather than their biological endowment as natural

kinds (Barrett, 2006a; 2006b; Russell & Barrett, 1999; Watson & Tellegen, 1985). For example,

according to the conceptual act model of emotion (Barrett, 2006a; 2006b), constructivists argue

that basic emotion labels (e.g., “fear” and “disgust”) have emerged from a common sense

tendency to conceptualize continua of subjective experiences into discrete categories, such as

describing the variety of the sky’s hues and shades as simply “blue”.2 Analogously, the inference

2As noted by the philosopher, Bertrand Russell (1912), the difference between the concept “blue” and the reality of

the sky’s myriad colours is, for practical purposes, unimportant. However, this difference is all-important to the

5

of the emotions “fear” or “disgust” as bottom-up natural kinds without accommodation of our

top-down conceptual forces would be a scientific error.

Relevant to my thesis, there is a parallel and overlapping categorical versus dimensional debate

within facial expression research. Opposing the view of facial expressions as having discrete

categories is the constructivist view that our facial expressions occupy dimensional continua of

possibilities, within which basic expression categories are oversimplified conceptualizations.

Accordingly, a criticism from the constructivists is that basic expressions are not immutable

readouts of our internal states, but they and other expressions become susceptible to higher-order

associations for social utility (Fridlund, 1997). Indeed, recent evidence has supported this view

that basic expressions are not strictly universal but the interpretation of their features depend on

culture (Jack et al., 2012) and social context (Aviezer et al., 2008; Aviezer, Trope & Todorov,

2012).

In light of these opposing views, facial expression researchers are confronted with a need to

reconcile the evidence and theory from both views.

One useful starting point is to acknowledge the physical breadth of our potential expression

space, a computational fact that cannot be debated. A conservative estimate of our possible

expression space can be defined by FACS as 20 separate action units, each of which may

contract at 5 different levels of intensity. For a single static expression, this amounts to one of 3.7

x 1016 possibilities. To put this number in perspective, it means that identifying a correct

expression in this space is the combinatorial equivalent of a person winning two consecutive

powerball lotteries.3

From the constructivist perspective, this vast possibility of potential expressions affirms the

continuous nature of a multidimensional expression space, whose complexity better captures the

variety and nuanced usage of our expressions, beyond six basic categories (e.g., Du, Tao, &

painter, who has to unlearn the common sense habit of how the sky seems in colour and learn the habit of seeing the

colours as they really appear. 3The odds of the winning the grand prize once is 1 in 1.75 x 108, and winning twice is its square, 1 in 3.1 x 1016.

6

Martinez, 2014). Simultaneously, from the categorical perspective, this provides a statistically

appropriate context for the evidence of cross-cultural consistency of basic expressions. If our

expressions were purely higher-order associations, socially learned for communication, there

could not be even the slightest recognition of expressions across cultures Ekman, Sorenson, &

Friesen, 1969. We would instead be left with arbitrary sets of expressions that are statistically

random and uninterpretable when viewed across cultures (such as the symbolic associations of

spoken languages).

What we require, then, is a theoretical framework that accounts for the dimensional variance of

our expressions and their higher-order communicative associations, while providing the

principles of how our common expressions are organized. Within this framework, basic

expressions would not be universal in the strictest sense but in having maintained statistical

stability across the myriad influences of culture and context they would indicate a common

ancestry. For such a parsimonious theory of facial expressions that unites both perspectives, we

turn towards Charles Darwin.

1.2 Facial Expressions: Theoretical Origins In his book, The Expression of Emotions in Man and Animals, published in 1872, Darwin posited

his theories on the origins of our emotions. He proposed three principles by which emotions may

be understood: the principle of serviceable associated habits, the principle of antithesis, and the

principle of direct action (or expressive discharge) of the nervous system. The first two of these

principles addressed how nature organized our expressions and are directly related to my thesis.

The first principle argues that expressions originated for some immediate egocentric functional

benefit, rather than their modern, allocentric communicative purposes. Thus an emotional

expression’s appearance is not arbitrary, but shaped in a way that was once congruently adaptive

with its emotion. The second principle of antithetical form argues that expressions can be

understood as originating from opposing actions. Thus, an expression may have another that is

opposite in appearance for an opposing function4. And because the face contains many of our

key sensory apertures (e.g., eyes, nose, mouth, ears), Darwin theorized that the function of

4It is worth noting that Darwin (1872) stated these opposing forms sometimes may not serve any function.

7

emotional expressivity was to adaptively modulate sensory intake, such as lowering of the brows

to reduce the eyes’ exposure to sunlight (Darwin, 1872).

These principles draw a nuanced but critical distinction between Darwin and the basic

expressions of Ekman and colleagues. While the categorical view was inspired by Darwin in

theorizing their common evolutionary origin, Darwin’s principles are less concerned with

expressions’ explicit categories and their immutability from higher cultural associations. Instead,

he placed emphasis on the bottom-up expressive features that once served the animal for some

sensory function (i.e., why they appear the way they do). Therefore the top-down semantic

categories that defined specific facial actions and their degrees of expressivity (i.e., what the

expressions are) would be left up to social interpretation (e.g., Aviezer, Trope, & Todorov, 2012;

Fridlund, 1997; Jack et al., 2012; Russell & Barrett, 1999).

From the Darwinian perspective, a basic expression such as “fear” represents not so much a

universal ideal but rather a probable grouping of facial actions that cohere toward some function

(e.g., vigilance towards threat in fear; Whalen, 1998). Framed in this way, the evidence for

cultural consistency (Ekman & Friesen, 1971; Ekman, Sorenson, & Friesen, 1969) may be

important as reference points that reveal how natural selection organized those expressive

features as probable tendencies rather than categorical ideals. Then, towards uncovering these

origins, basic expressions’ features are useful to consider as anchors, without which we would

find ourselves adrift in facial expressions’ combinatorial complexity. Further, in this vast space,

the inclusion of Darwin’s second principle of antithesis provides a potentially powerful

framework to position not just two opposing expressions but a way to align dimensional continua

of expression variance based on appearance and function.

Indeed, Darwin’s insights would be demonstrated empirically. In Susskind et al. (2008), we

found that by modeling the six basic expression appearances from a standard cross-cultural

expression dataset (Matsumoto & Ekman, 1988), there was, as Darwin theorized, a dimension of

opposing facial features that opened versus closed the sensory apertures—maximally opposed in

open fear versus closed disgust expressions. Testing the functional hypothesis of the potential

emotions underlying these expression forms, we found that opening the sensory apertures

enhanced the gathering of sensory information for fear’s theorized function of vigilance

8

(Whalen, 1998; Öhman & Mineka, 2001) while closing them reduced sensory gathering for

disgust’s function of rejection (Chapman et al., 2009; Rozin, Haidt, & McCauley, 2000).

1.3 Thesis: Sensory Evolution to Social Exaptation My thesis builds on Darwin’s principles of expressions’ origin. It embraces the categorical view

to the degree that basic expressions represent higher-order probabilities organized by lower,

adaptive actions. Thus, integrating the constructivist view into this framework, basic expressions

are not considered as universal but as tendencies of facial action that are scientifically useful in

navigating our expansive expression space. This provides a parsimonious explanation of facial

expressions’ modern role as social signals: rather than the emergence of completely new and

variant sets of expressive forms across different cultures, it is more likely that these invariant,

adaptive action tendencies were socially co-opted for communication (Andrew, 1963; Shariff &

Tracy, 2011).

The dimensional perspective of facial expressions is also part and parcel of this framework. We

know that our facial expressions can communicate a far greater variety than six basic emotional

states (Baron-Cohen et al., 2001; Baron-Cohen, Wheelwright, & Jollife, 1997; Du, Tao, &

Martinez, 2014), the full variance of which is better captured by dimensions rather than

categories of expressive forms (Oosterhof & Todorov, 2008; Plutchik, 1980; Rolls, 1990;

Russell, 1980; Russell & Barrett, 1999; Watson & Tellegen, 1985). Further, in accordance with

Darwin’s (1872) principles, the opening versus closing of the sensory apertures (Susskind et al.,

2008), provides an expressive dimension that can be used to specifically test this framework.

Taken together, my thesis is that the facial features that first evolved for sensory function for the

expression’s sender underwent social exaptation for communicative function for the expression’s

receiver. Based on previous work in Susskind et al. (2008), I use the facial features of fear and

disgust expressions as anchoring indices of the ends of a sensory opposition, which provides a

way to empirically test changes in sensory perception against the egocentric functions theorized

for the emotions of fear (Öhman & Mineka, 2001; Whalen, 1998) and disgust (Chapman &

Anderson, 2012; Rozin, Haidt, & McCauley, 2000). Building on a foundation of sensory

function, I then examine whether the same facial action tendencies were co-opted for social

function at different levels of information processing. At a basic level of information

transmission, I test the theory of social exaptation as whether the expressive features that confer

9

egocentric function also confer a congruent allocentric function by way of an enhanced physical

signal. Further, at the higher level of mental state communication, I test whether the invariant

features shaped for egocentric function may provide a parsimonious account of how we are able

to read a variety of our mental states from our facial expressions (Aviezer, Trope, & Todorov,

2012; Baron-Cohen et al., 2001; Baron-Cohen, Wheelwright, & Jollife, 1997; Du, Tao, &

Martinez, 2014; Jack et al., 2012).

1.3.1 The Eyes

I examined my thesis by specifically focusing on the eyes of facial expressions. This has the

benefit of not only controlling for the amount of facial expression information to reduce the

contextual confounds in their social interpretation (e.g., Aviezer et al., 2008; Aviezer, Trope, &

Todorov, 2012), but it also constrains the examination of expressions’ egocentric function to our

most studied and best understood sense, vision. Pertaining to expressive eyes of fear and disgust,

I examine the sensory functions of the eye widening features of fear and eye narrowing features

of disgust (Ekman, Friesen, Hager, 1978; Susskind et al., 2008). Specifically, according to the

emotion’s theorized function, fear’s eye widening is predicted to enhance sensory vigilance,

which can be empirically tested as improving visual detection and localization of potential

moving threats (Öhman & Mineka, 2001; Whalen, 1998). Conversely, disgust’s eye narrowing is

theorized to serve in visual discrimination (Sherman, Haidt, & Clore, 2012) toward a different

kind of threat, such as disease vectors and contaminated foods (Chapman & Anderson, 2012;

Rozin, Haidt, & McCauley, 2000).

Equally important is the communicative significance of the eyes. The ability of the eyes alone to

communicate a wide variety of mental states are not just familiar colloquialisms (e.g., “The eyes

are the windows to the soul”, “The eyes have it!”) but scientific evidence (Baron-Cohen et al.,

2001; Baron-Cohen, Wheelwright, & Jollife, 1997). Indeed, circumscribed brain regions in the

superior temporal sulcus and gyrus, which are responsive to eye information (Allison, Puce,

McCarthy, 2000; Calder et al., 2007), neighbor regions supporting how we read the mental states

of others (in the temporoparietal junction, Saxe & Powell, 2006). Convergently, increasing

failure to use the information conveyed by the eyes has been positively related with degrees of

autism, a disorder tied to failures in the ability to understand the expresser’s mental states

(Baron-Cohen, 1995).

10

1.3.2 Outline of Studies

To empirically test my thesis, I conducted studies of the egocentric and allocentric functions of

the eyes at different levels of information complexity. Studies in chapters 2 and 3 examine the

benefits of fear’s expressive eyes at a basic, physical signal level. Specifically, in chapter 2, I

examine whether eye widening in fear expressions provides the sensory benefit of widening the

expresser’s visual field. Such an increase of the visual field would aid in the identification of

potential threat, thus serving the egocentric vigilance function of the fear emotion (Whalen,

1998). Then in chapter 3, I examine whether this physical benefit for the expresser is passed on

as a physical benefit to the expression’s receiver by way of an enhanced directional eye gaze

signal. In this way, the benefits of fear’s eye widening in threat detection and localization is

directly passed on to our neighbours as a clearer “look here” signal. Critically, the studies show

that this can be traced to a physical enhancement without the communication of the fear emotion,

capitalizing on the unique morphology of the human eye, such as the white sclera that enhances

the physical gaze signal that is transmitted (Kobayashi & Kohshima, 1997). This provides

evidence of that expressions were co-opted as useful social signals at a basic physical level prior

to, or independent of, their use as signals of our inner mental states.

Studies in chapters 4 and 5 examine the benefits of the eye opening continuum for the expresser,

and how these features capture an important dimension in our ability to read complex mental

states from the eyes. In chapter 4, I examine how fear eye widening versus disgust eye narrowing

capitalize on a law of optics to provide a trade-off between a fundamental division in visual

function of sensitivity versus discrimination. Then in chapter 5, I examine whether the associated

facial features that confer this dimension of opposing sensory function has been socially co-

opted to convey various mental states of information sensitivity versus discrimination. In other

words, by uncovering a connection between the eye features that serve a specific perceptual

function with the communication of congruent mental states of information processing, I show

that our ability to read what the mind thinks may be based on how the eye sees.

11

2 Egocentric sensory benefits of expressive eyes5 As his first principle, Darwin (1872) theorized that facial expressions originated from a direct

egocentric benefit to the expresser as a precursor to their indirect allocentric signaling benefit to

the observer. To test Darwin’s first principle, I chose the eyes’ most salient feature—its wider

opening—and examined whether it confers a direct sensory benefit to the expresser.

A specific hypothesis about an expression’s egocentric sensory benefit requires a specific

expression. Our previous work found, through statistical modeling of the basic emotional

expressions, that the widest eye opening is characterized by fear (Susskind et al., 2008). Thus, I

examined whether eye widening in fear expressions originated for fear’s function of vigilance in

the face of uncertainty in the environment (Öhman & Mineka, 2001; Whalen, 1998).

Specifically, I hypothesized that eye widening in fear expressions would enhance the amount of

visual information gathered by enhancing the size of the expresser’s visual field, toward

detecting or locating potential threats.

To test the personal benefit of fear’s eye widening, I measured participants’ ability to identify the

orientation of controlled visual stimuli (Gabor patches; sinusoidal gratings enveloped by a

Gaussian) in the eccentric visual field, while they posed fear, neutral, and disgust expressions.

The eye widening of fear expression was contrasted with neutral and disgust expressions.

Disgust was selected as an emotional expression with the narrowest eye aperture, in opposition

to fear, thus spanning the largest possible variance in a continuum of expressive eye-opening

(Susskind et al., 2008). The use of disgust expressions also provided a control for the effort of

posing and holding a fear expression, which is not afforded by a relaxed neutral expression. It

also allowed the opposite sensory examination of whether eye narrowing reduces the visual field

relative to neutral.

An important point in my thesis is that expressive features are responsible for the adaptive

benefits through reconfiguring sensory intake, independently from the cognitive influences

emotions have at the level of the central nervous system (e.g., enhancing attention; Vuilleumier,

5Portions of this chapter have been published in Lee, Susskind, & Anderson (2013).

12

Armony, Driver, & Dolan, 2001). This is the reason for examining changes in visual perception

while participants posed fear or disgust expressions (Ekman, Friesen, & Hager, 1978) rather than

during the full-fledged experience of fear or disgust. This then requires testing the validity of the

posing procedure used on participants throughout my thesis, which I address now, before

describing this chapter’s main experiment.

2.1 General Methods and Materials

2.1.1 Participants

Predictions on facial expression are tested more robustly when taking into account the variance

of our individual differences in facial physical form (Blake, Lai, & Edward, 2003). All

participants in my dissertation were students from the University of Toronto, a highly

multicultural population that provided a heterogeneous sample of different eye anatomy.

2.1.2 Posing Facial Expressions and Validation

There are three experiments in my dissertation involving participants posing facial expressions

(Experiment 1 in chapter 2 and Experiments 5 and 6 in chapter 4). Prior to each experiment,

participants were instructed on the expressions’ Facial Action Coding System units (Ekman,

Friesen, & Hager, 1978). For fear: raising the eyebrows and drawing them together, opening the

eyelids, letting their mouth drop and stretch horizontally. For disgust: wrinkling the nose and

raising the upper lip for disgust. For neutral: relaxing their facial muscles. For Experiments 5 and

6, participants were trained to pose the expressions using only the upper face (for fear: raising

the eyebrows and drawing them together and opening the eyelids; for disgust: wrinkling the

nose; and for neutral: relaxing the face). This was due to the stability requirements of the chin

rests that were used. The experimenter provided coaching with demonstrations, mirrors, and

example images (Ekman & Friesen, 1976) that highlighted the action units.

The communicative validity of this directed expression task was tested in their ability to convey

fear and disgust. In chapter 3, I describe a set of schematic fear and disgust eyes, which were

created based on statistical modeling features (Cootes, Edwards, & Taylor, 2001) extracted from

a group of participants who posed fear and disgust expressions using these instructions (Susskind

et al., 2008). These schematic eyes were rated by an independent group of participants (in

Experiment 2B, chapter 3). Fear and disgust ratings of these fear and disgust eyes were

13

submitted to a 2 (rating) × 2 (expression) repeated measures ANOVA. Critically, a statistically

significant crossover interaction, F(1, 22) = 16.4; p < .001 ηp2 = .43, confirmed that fear eyes

were rated as more fearful than disgust and disgust eyes were rated as more disgustful than fear.

Next, I tested whether there were any potential cognitive influences from the facial feedback of

posing expressions. One measure of this is whether there are differences in the tendency to fixate

away from center to peripheral targets. In Experiment 5 (chapter 4), an automated eye-tracking

recorded the number of fixations away from center during posing fear, neutral, and disgust.

There, no differences were found between expressions, F(2, 20) = 0.03; p > .9. ηp2 = .003,

suggesting that the effect of expression was primarily sensory rather than a secondary effect of

emotional embodiment (Niedenthal, 2007).

Another measure of potential cognitive influence may be through autonomic feedback from

posing expressions (e.g., Strack, Martin, & Stepper, 1988). In Experiment 5, I measured pupil

size of participants as they posed expressions, as an index of autonomic feedback. Once again,

no pupil size differences were found between expressions, F(2, 20) = 0.13, p > .8, indicating a

lack of autonomic feedback. (See chapter 4 for details).

Together, these measures supported the validity of my posed expression paradigm. They

generated expressive eyes that were effectively perceived by independent observers as

emotional, while controlling for potential autonomic or cognitive effects from facial feedback.

Thus the experimental effects (in chapters 2 and 4) could be attributed to direct sensory

differences rather than indirect influences from facial feedback.

2.2 Experiment 1: Egocentric visual field In this experiment, to test the personal benefit of fear’s eye widening, participants posed fear,

neutral, and disgust expressions while making judgments of Gabor grating orientation in their

visual periphery. I hypothesized that fear would enhance identification farther out in the

periphery when compared to neutral or disgust.

Method

Twenty-eight undergraduates provided informed consent and participated for $10. All had

normal or corrected-to-normal vision.

14

To access the visual periphery, I projected a 1280 × 1024 resolution image (PC running E-Prime

1.1) calibrated to a size of 160 × 120 cm. The room was lit only by the projected image, and

participants were consented and trained in the room to control for any dark adaptation.

Participants were seated and stabilized via chin rest 30 cm from the projection screen fixated on

a 1 cm central cross at eye level. The fixation cross was drawn on a piece of tape and affixed to

the projection screen at each participant’s sitting eye level at the start of the experiment. This

was necessary as the shadow cast from the head occluded the projected central view. The

experiment program was then calibrated to this fixation height by the experimenter to automate

the experiment’s eccentricity calculations.

The design was a within-subjects factorial on expression (3) × meridian (3) × eccentricity (6).

Each trial (Figure 2.1a) began with a randomized delay of 700 to 1100 ms (in 100 ms

increments) followed by the target display for up to 4000 ms or until response. At target onset, a

400-ms tone was played over desktop speakers (500 Hz sinusoidal) because the target may not

have been visible. The target was one of four possible Gabor orientations (sinusoidal gratings

enveloped by a Gaussian at 0.5 cycles/deg, full contrast; created using Matlab 7.04; see Figure

2.1b). The Gabors appeared along one of three meridians (one vertical and two oblique; see

Figure 2.1a) at varying eccentricities from fixation, depending on calibration and practice phases

and always subtended 7.8° (the correct pixel-sized Gabor was selected based on eccentricity).

The horizontal meridians were omitted as no differences had previously been found across fear,

neutral, and disgust expressions, consistent with expressions only altering vertical eye aperture

(Susskind et al., 2008). Each trial ended with a 200 ms green or red feedback on the projected

screen indicating correct or incorrect/missed response, respectively (Figure 2.1a).

Trials were grouped in fours, with the experimenter instructing the participant to make and hold

a fear, neutral, or disgust expression. A group of neutral trials always followed a group of fear or

disgust trials to minimize fatigue. The experiment alternated between blocks of fear and disgust

of 48 trials per block (576 trials total; 144 fear, 144 disgust, and 288 neutral), with the starting

block counterbalanced across participants. Meridian was balanced within expression per block.

Eccentricity was balanced across blocks for each expression × meridian. Gabor orientation was

balanced within expression per block and balanced across eccentricities. Gabor phase was

15

randomized (one of four; ψ = 0°, 90°, 180°, 270°) but balanced within expression per block. A

break (up to 2 minutes) was provided halfway into the experiment.

Prior to the experiment, participants were trained on how to pose expressions (see instructions in

2.1.1). Participants then sat in front of a separate PC running E-Prime 1.1, 40 cm in front of an

LCD monitor displaying 1280 × 1024 (60 Hz) and performed 9 trials (3 per expression) of seeing

an exemplar image (taken from Ekman & Friesen, 1976) with arrows pointing to the instructed

action units, then holding the expression for 5 seconds.

Expression training was followed by response mapping training to the 4 Gabor orientations. This

was necessary because participants would not be able to look down at the keyboard during the

experiment. Participants performed a minimum of 48 trials, same in structure as the experiment,

except the target Gabor was presented centrally for the duration of the tone (400 ms) and a

separate response screen (showing the letters K, J, D, and F). For the first 24 trials, the response

screen showed the Gabor orientations corresponding to their keys (θ = 0°, 45°, 90°, and 135° to

K, J, D, and F, respectively). For the last 24 trials the letters did not appear, in order to ensure

participants could remember the response mapping. Trials were balanced and randomized up to

48 trials, after which additional trials were performed until a response accuracy criterion of 90%

for the last 20 trials was met.

Just before starting the main experiment (i.e., participant are now seated in front of the projection

screen) the experimenter executed a set of 9 trials (one for each expression × meridian) to

calibrate the range of eccentricities to test for each participant. In one trial, the participant was

instructed to pose and hold an expression, while the experimenter ran an informal

psychophysical staircase procedure, with the Gabor starting from the periphery and moving

toward the center along one of the three randomized meridians. The experimenter asked the

participant to identify the Gabor orientation on the keyboard. The orientation was randomized

each time the Gabor moved or after any key press. The Gabor was moved in larger increments to

start (several degrees) and then in 1° increments. If the participant responded correctly, the

Gabor was moved farther, and moved closer if responded incorrectly, repeating two to three

times until the experimenter felt that a threshold estimate was obtained. The procedure provided

rapid estimates of each expression × meridian’s threshold eccentricity. These were used to

determine the eccentricities to test in the main experiment (-6°, -2°, +1°, +3°, +6°, and +10° from

16

the estimate). After calibration, there were 48 practice trials (24 neutral, 12 fear, and 12 disgust,

divided into two blocks) that combined the training and reproduced the main experiment. These

trials tested 2 eccentricities closer than this estimate (-3° and -1°), as the goal was to familiarize

the participant with the trial structure. The starting block’s expression here matched the main

experiment. Calibration and practice data were excluded from analyses.

Figure 2.1. Egocentric visual field changes with expression. Time course of one trial is shown in (a) While maintaining fixation at the bottom of a projected image, participants posed and held a fear, neutral, or disgust expression. After a delay (randomized duration of 700 to 1100 ms), they judged the orientations of Gabor gratings (the 0° orientation is shown) presented along one of three meridians (the dashed lines showing these three meridians did not appear in the experiment). At target onset, a 400-ms tone was played over desktop speakers (500 Hz sinusoidal) because the target may not have been visible. As feedback, a bar (green for a correct response or red for an incorrect response) was displayed at eye level across the bottom of the screen. The four possible orientations of the Gabor stimuli (clockwise from top left: 0°, 45°, 90°, and 135°) are shown in (b). The graph in (c) shows results for the vertical meridian for 1 participant. The percentage of correct responses is plotted as a function of eccentricity of the Gabor stimulus for each of the three expressions; modeled logistic regression curves are also shown. For each combination of expression and meridian, each participant’s accuracy was computed at every eccentricity, and these values were then submitted to a separate logistic regression analysis with a single predictor (eccentricity) and a constant term. The lower bound was set to chance (25%). Psychometric threshold eccentricities (62.5%; marked by the dashed horizontal line in the graph) were computed for each logistic regression function. The rightward shift of the regression curves from disgust to neutral to fear indicates that successful stimulus identification extended farther out in the periphery. The bar graph (d) shows the mean psychometric threshold eccentricities for each of the three expressions. Error bars represent s.e.m.

17

Results

For each combination of expression and meridian, each participant’s accuracy was computed at

every eccentricity, and these values were then submitted to a separate logistic regression analysis

with a single predictor (eccentricity) and a constant term (see Figure 2.1c). The lower bound was

set to chance (25%). Psychometric threshold eccentricities (62.5%; marked by the dashed

horizontal line in Figure 2.1c) were computed for each logistic regression function.

Participants with eccentricity thresholds less than 0° or greater than 90° (suggesting an

inappropriate range of eccentricities sampled) were removed, leaving a final n = 19. The

threshold eccentricities for each participant’s expression were submitted to a repeated measures

ANOVA on expression (3) × meridian (3). As hypothesized, there was a main effect of

expression, F(2, 36) = 39.7, p < .001, ηp2 = .69 (Figure 2.1d), with participants identifying the

Gabor grating orientations at farther eccentricities when posing fear compared to neutral, F(1,

18) = 9.0, p < .01, ηp2 = .33, and disgust, F(1, 18) = 69.6, p < .001, ηp

2 = .80. Importantly, fear

expressions resulted in increasing the effective visual field by 9.4% into the available periphery

(maximum of 90°) compared to neutral. Disgust’s eye narrowing also resulted in the expected

visual field reduction compared to neutral, F(1, 18) = 31.6, p < .001, ηp2 = .64, associated with

its putative defensive function (Chapman & Anderson, 2012; Chapman et al., 2009; Rozin,

Haidt, & McCauley, 2000; Susskind et al., 2008).

Aside from the main effect of expression, there was a main effect of meridian, F(2, 36) = 59.8, p

< .001, ηp2 = .77, indicating that Gabor gratings were identified farther out for the oblique

meridians than the vertical meridian. This reflects the physiology of our horizontally elongated

eyes (Kobayashi & Kohshima, 1997) that only alter the vertical aperture, not the horizontal (this

was previously reflected in the lack of a visual field effect for fear and disgust along the

horizontal meridians; Susskind et al., 2008). There was no expression × meridian interaction,

F(4, 72) < 1.

2.3 Chapter Discussion Eye widening increased the effective visual field of the expresser. The results showed that for

identifying Gabor orientation wider fear eyes enhanced the expresser’s visual field 9.4% farther

out in the available periphery compared to neutral. This was a direct physical effect of reduced

18

occlusion of the upper visual regions by eye opening, as corroborated by the larger effect in the

vertical rather than oblique meridians (due to the morphology of our vertically-opening eyes) and

a visual field reduction for eye narrowing disgust expressions. Furthermore, the control measures

collected for validating this expression posing paradigm (described in 2.1.1) suggested this was

not due to an indirect influence from facial feedback.

These results corroborated a prior study where I showed similar effects of fear eye widening and

disgust eye narrowing on egocentric visual fields (Susskind et al., 2008). However, the present

experiment provides a more rigorous test of eye widening fear expression’s visual field

enhancement. Not only was the experiment here a psychophysical test over many more trials, the

Gabor stimuli used here were more complex and controlled for visual angle across eccentricities,

compared to the simple spot of light that was used previously (Susskind et al., 2008).

In accounting for disgust’s greater diminishment than fear’s enhancement when compared to

neutral (Figure 2.1d), it is worth noting that fear’s effects may be diminished in the directed

facial actions used here. In real fear, this egocentric sensory function is likely aligned with the

increase in sympathetic autonomic tone (Levenson, 1992) and the further conjunctive retracting

of the eyelids through the involuntary, sympathetically innervated Müller’s muscle (Brunton,

1938). Retraction of this muscle would provide greater peripheral visual field expansion for fear,

while having no analog for eye narrowing. Thus eye closure in disgust was likely more

pronounced than eye opening in fear. Additionally, considering the reduced acuity and contrast

sensitivity in the periphery (Cowey & Rolls, 1974; Rovamo, Virsu, & Näsänen, 1978), the

paradigm here of identifying targets may underestimate fear’s enhancement. Given the reduced

capacity for acuity in the periphery, fear expressions then likely promote enhanced detection

more than discrimination.

Another potential criticism toward expression’s egocentric sensory function is that the opposing

effect of visual field reduction from disgust’s eye narrowing provides no sensory benefit, rather

only a sensory hindrance compared to the benefit of eye widening fear. For now, one potential

explanation is that eye narrowing provides a distinct utility for the expresser toward closing off

the senses (Susskind et al., 2008). However, in chapter 4, I examine a more specific hypothesis

tied to my thesis that eye narrowing serves to benefit the expresser by enhancing visual acuity.

19

In summary, the egocentric sensory benefits of fear’s eye widening shown here supports an

origin in fear’s vigilance function (Darwin, 1872; Whalen, 1998) to gather immediate

information about potential threats in the environment. However, certain physical features that

are enhanced in fear’s eye widening provide no direct function for the expresser. For example,

the additional exposure of our physically salient white sclera (Kobayashi & Kohshima, 1997)

suggests an additional social function supported by expressive eye widening. In chapter 3, I

examine whether the egocentric sensory benefits shown here had a direct influence in shaping

their allocentric benefits—that is, how the single expressive action of eye widening, by

enhancing their physical saliency, may link the fear’s sensory function for the expresser directly

with that of the observer.

20

3 Social sensory benefits of expressive eyes6 While eye widening may have been shaped to directly modulate the expresser’s sensory intake

(Darwin, 1872; Susskind et al., 2008), the modern utility of our expressive eyes extends beyond

the self to serve as powerful social signals (Marsh, Adams, & Kleck, 2005; Whalen et al., 2004).

This is likely facilitated by the additional contrast provided by our white sclera, which is unique

among primates and is thought to have co-evolved with our social nature (Kobayashi &

Kohshima, 1997). Indeed, evidence has demonstrated that the eyes alone can convey rich

information about our innermost mental states (Baron-Cohen et al., 2001; Baron-Cohen,

Wheelwright, & Jollife, 1997), as well as signal where to direct our attention through eye gazes

(Friesen & Kingstone, 1998).

In this chapter, I examined whether the social utility of our expressive eyes were co-opted from

their older evolutionary function. That is, as fear’s eye widening conferred the egocentric

sensory benefit of identifying stimuli farther out in the periphery (chapter 2), I tested whether the

same eye widening provided an allocentric functional benefit to the observer at a basic signal

level, separate from the social perception (i.e., communication) of fear. Specifically, I examined

whether the observer’s judgment of the expresser’s gaze direction would be enhanced by eye

widening, such that the functional essence of expressive fear (in locating potential threat) would

be passed on to the observer through transmission of a clearer “look here” gaze signal.

Prior work has examined the interaction between fear and eye gazes, showing that fear

expressions facilitate faster judgments of averted gaze compared to direct gaze (Adams &

Franklin, 2009) and, inversely, that averted gaze enhances the perceived intensity of fear (Adams

& Kleck, 2005). Fear expressions’ directional eye gazes have also been shown to deploy

additional attention in the context of an attentional cueing paradigm (Putman, Hermans, & van

Honk, 2006; Tipples, 2006). While these effects illustrate how fear expressions benefit the

observer toward a congruent state of vigilance, it is important to note that they are hinged to the

communicated emotion—the state of alarm which may then reverberate in the observer to act as

the catalyst (e.g., Harrison, Singer, Rotshtein, Dolan & Critchley, 2006).

6Portions of this chapter have been published in Lee, Susskind, & Anderson (2013).

21

My examination here tested the interpersonally transmitted benefits of fear expressions at a more

basic, physical level. Retracting the eyelids and eyebrows has likely resulted from multiple

selective pressures, one of which may be to communicate a particular emotion or mental state

(Shariff & Tracy, 2011; Susskind et al., 2008). My thesis here is that one of these selective

pressures is that eye widening co-evolved to enhance processing of events in the expresser’s

visual field while simultaneously augmenting the gaze signal to aid the observer in locating the

same event. At a physical level, the conjunction of the increased iris-to-sclera contrast of the

human eye and its widening to enhance it may serve as the most expedient social signal of a

significant event’s location. Thus, the potential personal sensory benefit of eye widening would

be directly conferred interpersonally prior to and without the need for the communicated emotion

of the expresser.

However, it is important to note that fear expressions attract attention (Vuilleumier, Armony,

Driver, & Dolan, 2001), which may bias performance. In order to limit the influence of the

perceived emotion from the physical signal of the eyes without altering their physical form, in

Experiment 2A, I created simple schematic outlines of just the eyes extracted from full fear

expressions (Figure 3.1b). In Experiment 2B, I further examined their effects when inverted,

hypothesizing that perceived fear would be reduced (McKelvie, 1995) while maintaining the

stronger gaze signal. This would provide support for the notion that the physical signal rather

than perception of fear is responsible. Finally, in Experiment 3, I used the same eye gazes in an

attentional cueing paradigm (e.g., Friesen & Kingstone, 1998) to test whether gaze signal

enhancement facilitated locating peripheral targets, while simultaneously examining their

capacity to alter covert attention. I begin by detailing the eye gaze stimuli used in these

experiments.

3.1 General Methods: Eye Modeling I created schematic eye stimuli, without the rest of the face, to impoverish the emotional

influence of the full expression while retaining the basic physical features (Figure 3.1b). To this

end, the eye shapes were created using a statistical model (Cootes, Edwards, & Taylor, 2001) of

photographed posed fear and disgust expressions from 19 participants from a previous

experiment (Susskind et al., 2008).

22

For gaze direction judgment (Experiments 2A and 2B), I used the eye models to compute

average fear and disgust eyes, along with two intermediate sizes (Figure 3.1b). The averaged fear

and disgust eyes were scaled by 1.25 to mitigate some of averaging’s dampening effects on the

facial features. The eyes were symmetric (mirrored using left eyes of the exemplars), with eye

width, distance between eyes, and iris diameter constant across all eye stimuli, all averages of the

exemplars. Straight gaze eyes had each iris horizontally centered on its eye. Eye stimuli were

anchored on straight gaze iris positions such that only eye outlines, never irises, changed their

vertical positions while only irises, never eye outlines, changed their horizontal positions. Matlab

7.04 was used to compute the model and create the eye outline curvature using splines. The

remaining fine tuning, controlling for individual differences and iris placement, were performed

using Adobe Illustrator.

Lastly, I independently tested whether these schematic fear eyes indeed impoverished the

perceived emotionality compared to realistic fear eyes. I compared participant ratings of these

eyes (from Experiment 2B) to ratings collected from a separate experiment where participants (n

= 28) rated photographic images of fear eyes generated using a similar statistical model (for

details, see chapter 5, Experiment 6). Independent t-tests confirmed that schematic fear eyes

were rated as less fearful (t(49) = 3.1, p = .003) and less accurately recognized as fear among the

6 basic emotions (67.4% compared to 86.4%; t(49) = 3.7, p < .001). Recognition accuracy was

computed for each participant as the percentage of trials fear was rated higher than the five

remaining basic emotions (disgust, anger, happy, sad, and surprise), with chance at 50% and ties

scored as 0.5.

3.2 Experiment 2A: Gaze Direction enhancement In this experiment participants judged the direction of eye gazes across 4 different eye sizes,

from narrowest disgust to widest fear (Figure 3.1b). I hypothesized that wider eye gazes would

provide a stronger “look here” signal and thus be judged more accurately.

Method

Twenty-six undergraduates provided informed consent and participated for $10 or course credit.

All had normal or corrected-to-normal vision.

23

Participants were seated and stabilized via chin rest 40 cm from an LCD monitor, displaying

1280 × 1024 (60 Hz) from a PC running E-Prime 1.1. Participants fixated on a 0.2° central cross

and responded to a target pair of eyes that subtended 7.2° × 0.47° (smallest) to 1.0° (largest). For

each eye size, eleven degrees of gaze stimuli were created by shifting each iris from its center by:

L0.25°, L0.13°, L0.063°, L0.031°, L0.016°, 0°, R0.016°, R0.031°, R0.063°, R0.13°, and R0.25°.

The experiment was a within-subjects factorial design on eye size (4) × degree of gaze (11).

There were 528 trials balanced across four blocks and randomized. Each trial (Figure 3.1a) began

with a randomized fixation period of 1000 to 1400 ms (in 100 ms increments), followed by a pair

of eyes and fixation cross for 300 ms, then the rectangular mask (11.1° × 2.8°) for up to 2000 ms

or until response. Participants responded during the mask to whether they perceived the eyes

gazing left or right with their respective left or right index fingers in a two-alternative forced-

choice (2AFC) task. A central red square feedback appeared for 200 ms if no response was given

to keep participants on task (Figure 3.1a).

The red feedback square subtended 0.9°. All stimuli appeared on a 75% gray background. Breaks

(up to 1 minute) were provided halfway through Experiment 2A and between each quarter in

Experiment 2B.

There were 12 practice trials of a unique set of eyes using the two farthest degrees of gaze for

each direction (L0.25°, L0.13°, R0.13°, and R0.25°).

24

Figure 3.1. Gaze direction enhancement with wider eye opening. Time course for one trial is shown in (a). Each trial began with fixation (randomized duration of 1000 to 1400 ms). An eye-gaze stimulus then appeared for 300 ms, after which it was masked. The rectangular mask appeared for up to 2,000 ms or until the participant responded to indicate the eyes’ gaze direction. A red square appeared for 200 ms as feedback if the participant failed to respond. Stimuli consisted of schematic drawings of differently sized eyes (b), which were modeled using participants who were instructed to pose disgust expressions (top images; Size 1) and fear expressions (bottom images; Size 4). Intermediate Sizes 2 and 3 were interpolated linearly from Size 1 to Size 4 in equal steps of vertical aperture. Eyes in the right column are inverted versions of eyes in the left column and were used only in Experiment 2B. All degrees of gaze shown are 0.063° to the left of center. The graph in (c) shows the pattern of correct gaze-judgment responses as a function of degree of gaze (L = left of center, R = right of center) and eye size in Experiment 2A (each circle represents the average of all participants across all trials at a given degree of gaze); modeled logistic regression curves are also shown. The steeper logistic regression slope for fear eyes compared with disgust eyes indicates greater gaze-direction discriminability. The graph in (d) shows a plot of mean maximum logistic regression slopes computed using separate logistic regression models for each eye size for each participant. Slope values were standardized for the upright and inverted conditions within each experiment (Exp.). Close overlap across experiments indicates replicated effects of increased judgment accuracy with increased eye size. Error bars represent standard errors of the mean.

Results

Prior to the main analysis, I examined the overall fidelity of gaze direction judgment. Notably,

participants correctly identified the smallest degree of gaze from center (L0.016° and R0.016°) at

62.7%, significantly greater than chance (t(25) = 8.6, p < .0001).

The 2AFC data (left coded as 0; right as 1) from all participants were submitted to a logistic

regression model of three predictors (degree of gaze, eye size, and degree × size interaction;

Collett, 1991) and a constant term. Logistic regressions were performed with the lower and upper

25

bounds (i.e., [1-λ], where λ = lapse variable; Wichmann & Hill, 2001) set at the response

accuracy at the farthest degree of gazes (L0.25° and R0.25°) across all eye sizes and participants.

This average differed slightly across experiments. In Experiment 2A, this average was 0.950

(i.e., λ = 0.050) and the lower and upper bounds were set at 0.050 and 0.950, and in the

following Experiment 2B it was 0.975 (i.e., λ = 0.025) and the bounds were set at 0.025 and

0.975.

Degree of gaze was a highly statistically significant predictor of left/right responses (Likelihood

Ratio = 6780.0, p < .0001) indicating that response accuracy improved as degree of gaze shifted

away from center to either direction. Eye size alone was not significant in predicting left/right

responses (Likelihood Ratio = 0.41, p > .5). Critically, the degree × size interaction was

statistically significant (Likelihood Ratio = 23.8, p < .0001), indicating that changes in eye size

exerted different logistic regression curves. Specifically, the logistic regression curve slopes

were steeper as eye size increased (Figure 3.1c,d), supporting my hypothesis that participants

were better able to determine gaze direction as eye size increased.

3.3 Experiment 2B: Physical versus emotional eye signals Despite using schematic eyes, the larger eyes may have still expressed some degree of fear,

which may have altered attention and direction judgment accuracy. To examine this possible

influence, I conducted the same experiment while including a set of the same eyes inverted

(Figure 3.1b), which provide exactly the same amount of physical gaze information. I

hypothesized that the change in eye configuration (e.g., fear’s greater exposure of white sclera

now below the iris) would reduce the perception of fear in the larger eye sizes, as measured by

subjective ratings but would leave enhanced gaze direction judgment intact.

Method

Twenty-five new undergraduates provided informed consent and participated for $10. All had

normal or corrected-to-normal vision. All aspects of the experiment were the same as

Experiment 2A except for the addition of 4 new pairs of vertically inverted eyes, thus doubling

the number of trials (1056) as a within-subjects factorial design on eye size (4) × degree of gaze

(11) × inverted (2).

26

Afterwards, participants rated all 8 pairs of eyes in how much they express each of the 6 basic

emotions. Each pair of eyes (gazing straight) was shown in random order, then rated from 1 (Not

at all) to 9 (Very strongly) on the basic emotion labels presented one at a time: anger, disgust,

fearful, happy, sad, and surprise. Participants were instructed that there were no right or wrong

answers and that a pair of eyes may appear to express multiple or no emotions.

After the ratings, participants were asked by the experimenter whether they realized some of the

eyes were inverted versions of the upright eyes.

Results

One participant recognized that the eyes were inverted. Another participant only responded to

66% of the trials in the main experiment. These data were removed from analysis (final n = 23).

The ratings of fear were submitted to a repeated measures ANOVAs on eye size (4) × inversion

(2). There was a main effect of eye size, F(3, 66) = 9.0, p < .0001, ηp2 = .29, as increasingly

wider eyes were perceived as expressing more fear, F(1, 22) = 18.2, p < .001, ηp2 = .45.

Critically, a main effect of inversion, F(1, 22) = 9.6, p < .01, ηp2 = .31, confirmed that inverted

eyes were rated as expressing less fear than upright eyes. For example, inverted eyes of size 3

were perceived to express as much fear as upright eyes of the smallest size 1, t(22) = 0.04, p > .9

(Figure 3.2a). Despite the similar emotional perception between these two sizes, gaze direction

of inverted size 3 eyes were perceived more accurately than size 1 (Figure 3.2b).

The gaze judgment data for all participants were split into upright and inverted sets and

submitted to two separate logistic regression models, each identical to Experiment 2A.

Replicating the finding from Experiment 2A, there was a degree × size interaction for upright

eyes (Likelihood Ratio = 26.7, p < .0001; Figure 3.1d). As hypothesized, gaze judgment

enhancement was unaffected by emotionality, as seen in a similar, and numerically greater, effect

of degree × size for inverted eyes (Likelihood Ratio = 28.6, p < .0001; Figure 3.1d). When the

full dataset was analyzed as one logistic regression model with the inverted condition as an

additional parameter, it did not significantly modify degree of gaze (neither interactions

significant: degree × inverted, Likelihood Ratio = 0.18, p > .6; degree × size × inverted,

Likelihood Ratio = 0.04, p > .8).

27

Figure 3.2. Compared effects of upright and inverted eyes. (a) Size 3 inverted eyes are perceived as fearful as narrowest Size 1 upright eyes. (b) Despite similarity in perceived fear, the steeper logistic regression slope indicates Size 3 inverted eye gazes were judged more accurately than Size 1 upright eyes.

3.4 Experiment 3: Peripheral target localization I next examined whether fear’s eye widening would also facilitate observer responsiveness in

locating peripheral stimulus events (i.e., to “look here”), thus potentially extending the benefit of

a clearer gaze signal transmission. I used the same schematic eyes in an attentional gaze cueing

paradigm, where fear expressions have shown to direct more attention to cued locations than

neutral gazes (Putman, Hermans, & van Honk, 2006; Tipples, 2006). If enhanced gaze direction

judgment reflects a special capacity for these eyes to interact with attention, larger eye sizes

would interact with attentional cueing effects. On the other hand, a lack of interaction would

make a stronger case for a purely physical gaze enhancement. I hypothesized the latter based on

my thesis and prior evidence of an attention interaction that depends on the communicated

emotion (i.e., using full fear expressions, with the effect related to anxiety traits; Fox, Mathews,

Calder, & Yiend, 2007; Putman, Hermans, & van Honk, 2006).

In addition, using the same statistical expression model (described in Section 3.1), I maintained

individual differences in eye shapes to create 4 eye sizes for each of the 19 exemplars (see Figure

A1) to generate broader variances in measurable eye features (Figure 3.3a). I then related these

features to response time to explore which might be the most influential functional features in

eye widening.

28

Method

Twenty-six new undergraduates provided informed consent and participated for $10 or course

credit. All had normal or corrected-to-normal vision. The setup and apparatus were identical to

Experiment 2A.

Participants fixated on a 0.45° central cross and viewed one pair of the schematic eyes (4 sizes ×

19 exemplars), subtending 14.4° × 0.27° to 2.3°, gazing straight, then either left or right (L1.0°

or R1.0°). Participants then responded with their respective left or right index fingers (pressing

either the Z or ‘/’ key) to the location of a target asterisk (0.55°), which appeared beside the eyes

(8.6° left or right from fixation), congruent or incongruent with gaze direction. A green or red

feedback, subtending 0.9°, appeared at the end of each trial.

The design was a within-subjects factorial on cue congruence (2) × eye size (4) × exemplar (19).

There were 304 trials balanced and randomized across four blocks on eye size and congruence.

Each trial began with a randomized fixation period of 700 to 1100 ms (in 100 ms increments),

and participants were instructed to fixate on this location throughout the trial. The cross was

replaced by a pair of eyes gazing straight for 1000 ms. The eyes then shifted gaze either left or

right for 200 ms and held their gaze until the end of the trial. The target asterisk appeared after

the 200 ms of gaze onset and remained on screen for up to 800 ms or until response. A central

green (for correct) or red (for incorrect or no response) square feedback appeared for 200 ms to

keep participants on task. All stimuli appeared on a white background. A break (up to 1 minute)

was provided halfway.

There were 8 practice trials using a unique set of eyes, one trial per each cue congruence × eye

size condition.

Specific eye features were computed in Matlab 7.04. Eye aperture was computed as the pixel

distance between the top and the bottom of the eye (always at the eye’s horizontal center, even

when the corners of the eyes were lower). The other eye features measured pixel counts and

luminances that were inside the eye outlines: number of black pixels for iris area; number of

white pixels for sclera area; and standard deviation of all pixel luminances for contrast.

29

Determining the vertical locations of the irises was less clear, as many iris positions were

difficult to determine from the original photographs, especially for narrow-eyed disgust, and the

two intermediate sizes which were computationally generated. For consistency, I applied an

arithmetic rule to all the irises, based on the available fear and disgust photographs. The results

were slightly varied, however, with a tendency for the larger eye sizes to be looking downward

and the smaller eyes looking upward (Figure A1). Note that, as in Experiments 2A and 2B, the

absolute vertical iris positions did not change across eye stimuli but their relative positions

within the eye outlines produced this effect. A concern was that this vertical gaze effect may not

have cued the ideal locations of the asterisk targets (i.e., instead cueing above or below it). Thus

in order to independently control for this effect, I ran a short experiment afterward where

participants rated whether they perceived these gazes to have a “vertical component” of either

up, center, or down.

This experiment consisted of 76 randomized trials, one for each eye pair used in Experiment 3

(Figure A1). Each trial began with a randomized fixation period of 700 to 1100 ms (in 100 ms

increments), followed by one of the 76 eye stimuli, gazing left or right. Participants responded

with their dominant hand: numpad 8, 5, or 2 keys if right-handed or W, S, or X keys if left-

handed indicating up, center, or down, respectively. The gaze stimulus remained on the screen

for 1500 ms or until response. No feedback was given. Gaze direction was randomized as left or

right, balanced across the eyes. There were 8 practice trials, using randomly selected eyes from

the stimulus set, balanced across sizes.

Vertical gaze responses were coded as 1, 0, and −1 (for up, center, and down, respectively) and

ratings for each of the 76 eye pairs were averaged across all participants. There was a highly

significantly correlation between vertical rating and eye aperture, r(74) = −.869, p < .0001,

confirming that as eye aperture increased from small to large, participants saw the vertical gaze

also shifted from looking up to looking down. This relationship between vertical gaze ratings and

eye aperture suggested it should be controlled for in the analysis of eye features and response

time.

30

Figure 3.3. Revealed iris area predicts response time to correctly cued peripheral targets. The illustration (a) shows the four features extracted from each unique pair of schematic eyes. The graph (b) shows the relationship between the visible iris area revealed by eye widening and response times for correctly cued targets. Axes represent residuals, after controlling for baseline vertical-gaze perception using data collected in a separate experiment.

Results

Two participants were removed as outliers in target accuracy (49.7% and 69.7%; final n = 24).

Response times less than 150 ms were removed from analysis as well as response times longer

than 2 standard deviations from each participant’s mean.

Mean response times for only the correctly responded trials were submitted to a repeated

measures ANOVA on cue congruence (2) × eye size (4). Consistent with an effect of attention,

there was a main effect of cue congruence, F(1, 23) = 42.1, p < .0001, ηp2 = .65, with

participants responding faster to congruent than incongruent cues, replicating previous gaze

cueing findings (Friesen & Kingstone, 1998). The congruence × size interaction was not

statistically significant, F(3, 69) = 0.83, p > .4, ηp2 = .035, indicating that eye widening did not

modify this attentional cueing effect. However, there was a main effect of eye size, F(3, 69) =

7.98, p = .0001, ηp2 = .26, with increasing eye widening facilitating responses to peripheral

targets, F(1, 23) = 12.7, p < .01, ηp2 = .36.

My thesis within this chapter is that enhanced gaze direction decoding is brought about by

changes in the physical parameters of the iris and sclera in widened eyes. To explore this

relationship, I first extracted four features for all 76 pairs of eyes: vertical aperture, contrast,

visible area of iris, and visible area of sclera (Figure 3.3a). I then ignored eye size groups and

correlated each feature with the congruently cued and correctly responded trial response times,

averaged across all participants. I used congruent gaze cues because they are more functionally

31

plausible toward transmitting useful cues and involve simpler processes than incongruent cue

responses, which include the investment and divestment of directed attention (Friesen &

Kingstone, 1998).

Controlling for the vertical directionality gaze effect (see above) improved all eye feature

correlations suggesting that it was a significant mediating factor. After controlling for this effect,

I found iris area to have the largest correlation to faster responses (r(72) = −.378, p = .001;

Figure 3.3b), followed by contrast (r(72) = −.339, p = .003). Of the 8 correlations (4 features

without controlling for vertical iris position and 4 features with), only these two features (iris

area and contrast controlled for iris position) survived the multiple comparisons correction of: 1

− (1 − α)1/8 = .00639; α = .05. Given these two variables’ close correspondence (r(72) = .683, p

< .0001), they are likely driven by the same variable.

3.5 Chapter Discussion Eye widening enhanced the discriminability of eye gazes for the observer and facilitated the

speed of using gaze cues to locate peripheral stimuli. I showed that this was largely due to an

augmented physical signal of the eyes, without the fearfulness of the complete expression, and

retained under conditions of eye inversion, where fear perception was significantly diminished.

An analysis of eye features suggests the mechanism may be due to increased iris exposure and

associated increase in local iris-to-sclera contrast found in wider eyes. Based on these results I

argue that expressing fear, because of its characteristic eye widening, provides a functional

transfer from sensory to social, toward locating a common environmental event.

The social importance of directional eye gaze is signified in the unique morphology of our eyes,

seemingly evolved to maximize its physical signal by the additional contrast of our white sclera

that likely facilitates observers’ ability to determine gaze direction (Kobayashi & Kohshima,

1997). The social utility of this information is indicated by the separate representations of left

and right directional gazes in the superior temporal sulcus (Calder et al., 2007) that are proximal

to a region important for the decoding others’ mental states (Saxe & Powell, 2006). How aptly

we process this information can be seen behaviorally, in the accurate perception of minute

changes in eye gaze (e.g., the smallest shift of gaze here, 0.016° or one-half pixel in our

experimental setup, possible through anti-aliasing, was reliably discriminated above chance

despite its brief, masked display), and our remarkable sensitivity to different contexts of head

32

orientation in correctly discriminating gaze direction (Jenkins, Beaver, and Calder, 2006;

Langton, Watt, & Bruce, 2000).

Given that mere greater exposure of eye whites can activate the amygdala (Whalen et al., 2004)

and widened eyes are sufficient to recognize fear (Smith, Cottrell, Gosselin, & Schyns, 2005),

the recruitment of emotional circuitry, as well as some degree of emotion contagion (Harrison et

al., 2006) to the wider eyes is possible. Thus, some emotional modulatory influence on gaze

processing in the wider eyes cannot be completely ruled out. Towards support of the physical

signal argument here, I tested the same eyes inverted, as inverted fear expressions have

demonstrated reductions in amygdalar activity and attentional orienting (Bocanegra &

Zeelenberg, 2009; Phelps, Ling, & Carrasco, 2006; Sato, Kochiyama, & Yoshikawa, 2011)

(although they have been shown to retain some of their affective influences; Lipp, Price, &

Tellegen, 2009). These inverted eyes resulted in a pronounced reduction in fear perception

(Figure 3.2a) but exerted an equivalent pattern of gaze discrimination enhancement (Figure

3.1d), consistent with their dissociation. While it remains possible that inverted eyes still resulted

in some emotional response in the perceiver, it is unclear how distinct emotional responses may

have enhanced (fear) versus impaired (disgust) gaze discrimination. Rather, the most

parsimonious account is the evident difference in physical signal originating from the iris and

sclera.

Eye widening in emotional expressions may reflect some evolutionary or learned adaptation to

enhance the physical signal of the eye. Fear and disgust both communicate emotional states, thus

being equal potential targets for attention, but the salience of the signal transmitted by their eyes

is biased towards fear. This asymmetric salience of eye widening may also enhance detection of

these expressions from afar (Smith & Schyns, 2009), in addition to transmitting a clearer “look

here” signal, both aligned with fear’s selective pressures towards rapid processing of distal,

moving threats, as compared to disgust which tracks proximal, stationary ones (Anderson et al.,

2003). That is, fear and disgust are both threat responses of different kinds, with the former

appropriate for a potential predator or looming threat and the latter for a potential disease vector

(e.g., Chapman et al., 2009).

The asymmetry in the kinds of threat fear and disgust respond to may explain the asymmetry of

disgust’s uncertain interpersonal physical signaling benefit. Certainly, at higher orders of social

33

utility than the physical signaling examined here, facial expressions may serve more complex

social intents (Fridlund, 1997), decoupled from their emotional communicative signals. Thus,

modern social utility of disgust’s narrow eyes may serve more complex social functions, such as

the signaling of morally repulsive individuals to its onlookers (Chapman & Anderson, 2012;

Chapman et al., 2009).

However, it is also possible that opposition in form (e.g., fear versus disgust) does not strictly

obey opposition in egocentric or allocentric function (Darwin, 1872)—as how disgust eye

narrowing only reduces the expresser’s visual field in chapter 2 and hinders physical signaling

for observers here. In the next chapter, I explore eye opening more fully, as a continuous

dimension of facial appearance and sensory acquisition, the opposing ends of which have

opposing sensory benefits for the expresser (Chapman et al., 2009; Darwin, 1872; Susskind &

Anderson, 2008; Susskind et al., 2008).

34

4 Optical origins of expressive eye widening7 As his second principle, Darwin (1872) theorized that emotional expressions originated as

opposing actions along some varying dimensions (Oosterhof & Todorov, 2008; Russell &

Barrett, 1999; Susskind et al., 2008). Here, I tested Darwin’s second principle as related to the

expressive dimension of eye opening. Specific to my thesis, I examined whether opposing

expressive features that widen and narrow the eyes were selected to functionally benefit the

expresser by exploiting a physical principle of how light refracts.

Although facial muscles that reconfigure superficial eye features should have no direct influence

on the pupil or the accommodative lens behind it, approximately two thirds of the eye’s full

refractive power comes from the cornea (Duke-Elder & Abrams, 1970). I therefore hypothesized

that facial expressive behaviors that expose or conceal the cornea would have measurable

consequences on the eye’s optics and thereby perception. Specifically, I hypothesized that

widening the eyes would increase the gathering of light to enhance sensitivity at the expense of

acuity, thus prioritizing visual detection over discrimination. Conversely, I hypothesized that

narrowing the eyes would better focus light to enhance visual acuity at the expense of sensitivity,

thus prioritizing visual discrimination over detection.

The functional basis of this sensitivity-versus-acuity opposition is a familiar one, seen as a

fundamental division throughout the visual system. Starting from retinal rods and cones, the

magnocellular and parvocellular systems (Livingstone & Hubel, 1987) represent a fundamental

trade-off between sensitivity and acuity carried on to the dorsal and ventral streams for the

processing of “where” and “what” information, respectively (Ungerleider & Mishkin, 1982). My

specific thesis here is that the expressive dimension of widening versus narrowing the eyes arose

from a need to filter information toward one of these two channels, thus enhancing either the

gathering or focusing of light to modulate the ability to detect or discriminate stimuli,

respectively, in a situation-appropriate manner. After some initial coarse appraisal, an expression

would serve to modify perceptual encoding toward one of two opposing needs—to increase

sensitivity or discrimination.

7Portions of this chapter have been published in Lee, Mirza, Flanagan, & Anderson (2014).

35

Once again, I selected expressions of fear and disgust to test this hypothesis because they

represent the largest and smallest eye apertures, respectively, consistent with their structural

opponency (Susskind et al., 2008). Once triggered by a cue or context (e.g., faint movement,

sounds, or odours; Öhman & Mineka, 2001), eye widening may improve detection and

localization of a potential threat that requires enhanced vigilance, which would be consistent

with the hypothesized function of fear (Whalen, 1998). Conversely, eye narrowing may improve

perceptual discrimination (Sherman, Haidt, & Clore, 2012) to discern different kinds of threats,

such as disease vectors and contaminated foods, avoidance of which is a hypothesized function

of disgust (Chapman & Anderson, 2012; Rozin, Haidt, & McCauley, 2000).Rather than treat

these as distinct emotions with distinct functions, however, I hypothesized that their functions

are opposed along a single continuum, in accordance with Darwin’s (1872) principle of

antithesis. This also allowed me to address the issue raised in chapter 2 of whether eye narrowing

in disgust has a sensory enhancement function, rather than a mere sensory reduction (Susskind et

al., 2008).

To examine these hypotheses, I first created a basic optical model of how sensitivity and acuity

might be affected by facial expressions by using eye aperture measurements of participants who

posed expressions of fear and disgust. I then tested the model’s predictions by measuring

sensitivity (Experiment 4) and acuity (Experiment 5) in separate groups of participants using

standard measures of visual function—the Humphrey Field Analyzer (HFA; Carl Zeiss Meditec,

Dublin, CA) for sensitivity and Bailey-Lovie eye charts (Bailey & Lovie, 1976) for acuity.

4.1 Optical Model To create the optical model, I used posed fear and disgust expressions gathered from 19

participants in a prior study (Susskind et al., 2008; described in chapter 2.1.2). One image frame

from each posed expression was submitted to appearance modeling (Cootes, Edwards, & Taylor,

2001). The image frame was then scaled, rotated, and aligned in Matlab Version 7.04. The left

eye’s aperture (distance from the top to the bottom of the eye) from each expression was

extracted for use in this optical model. The apertures for each participant’s fear and disgust

expressions were averaged to calculate a theoretical neutral.

36

4.1.1 Sensitivity

Sensitivity can be defined as the amount of light gathered by the eye because increased gathering

enhances the likelihood of detecting the light source. The amount of light gathered is directly

related to the area of the exposed eye by which light is collected and refracted through the

cornea, eventually arriving on the retina. For simplicity, this model assumed a constant pupil

size, sufficiently open that extra light refracted by greater exposure of the cornea would not be

blocked by the small pupil. The exposed eye area was the measure of sensitivity in this model.

Because human eyes are horizontally elongated (Kobayashi & Kohshima, 1997) and open

vertically rather than concentrically, I approximated the area of eye opening through which light

is collected as a simple rectangle with a constant width (w) and a height equal to the eye aperture

(corneal diameter, d; Figure 4.1a). This area was the model’s index of sensitivity:

Because width remained constant for each participant, sensitivity was solely a function of height.

Sensitivity was thus predicted to increase as eye aperture increased from disgust to fear. To

compute the model’s predicted sensitivity values (Figure 4.1c), each participant’s eye width was

set equal to his or her eye aperture height for fear.

4.1.2 Acuity

Acuity can be defined as the ability to discriminate two proximate points of light. If the points

are imperfectly focused by the eye (e.g., because of nearsightedness), they arrive on the retina as

blurs that overlap and hinder discriminability. Photographers call these blurs circles of confusion

(CCs). I use the same term, although the blur is technically not a circle here because the eye

opens vertically not concentrically. Because discriminability is impaired with larger CCs, acuity

was indexed in this model as the negative size of the CC. Assuming constant pupil size as before,

I approximated the eye as a single thin lens of focal power f0. A cone of light rays from a point

light stimulus in front of the lens, at distance s, travels through the eye aperture, d, and is

correctly focused as a point at distance v0, behind the lens, on the retina (Figure 4.1b, black

lines). When correctly focused, no blur is created with eye widening or narrowing. However, if

focal power is imperfect (i.e., modeled as f1 as a result of nearsightedness), the same light is

37

incorrectly focused at point v1 and falls on the retina instead as a CC of size cF (Figure 4.1b, red

lines). From the thin-lens equation and similar triangles results the following equations:

Algebraically combining these equalities results in the CC size formula:

Then, if the eye aperture is reduced to that of disgust, dD, the CC size is also reduced, cD (Figure

4.1b, blue lines), by affecting only the numerator:

That is, CC size is directly proportional to eye aperture, and because acuity is the opposite of CC

size, acuity is computed as its negative:

Acuity was thus predicted to improve as eye aperture decreased from fear to disgust. To compute

model values, the light stimulus was set at a constant distance, s, of 6 m. I used an approximate

human-eye focal length, f0, of 22 mm, and then modeled three nearsighted conditions of +1.0,

+2.0, and +3.0 diopters (f1 = 21.53, 21.07, and 20.64 mm, respectively), congruent with

38

conditions in the acuity experiment (Experiment 5). I then averaged acuity across the diopter

conditions to compute the acuity that the model predicts (Figure 4.1c).

Figure 4.1. Modeled optical effects of eye aperture. The area of eye aperture through which light is collected was approximated as a rectangle of constant width (w) and varying eye aperture diameter (d), as shown in (a). This diameter is larger for fear (dF) than for disgust (dD). In (b) the black lines show a cone of rays from a light source, s, correctly focused at the retina, v0, by the lens of focal power f0. When correctly focused, no blur is created with eye widening or narrowing. But with an imperfect lens of focal power f1, as in nearsightedness, the light is focused too close, v1, and arrives on the retina as a blurred circle of confusion of size cF (red lines). With a narrower aperture, dD, light rays are focused at the same point (v1) but create a smaller circle of confusion of size cD (blue lines). The graph in (c) shows model predictions of mean sensitivity (left y-axis) and acuity (right y-axis) varying with facial expression. Sensitivity scores are indexed by exposed eye aperture area, as illustrated in (a). Acuity scores are indexed by negative size of the circle of confusion, as illustrated in (b). Higher scores indicate greater sensitivity or acuity. Error bars represent s.e.m.

4.2 Experiment 4: Measuring Sensitivity Method

To test the model, I measured the left-eye sensitivity of 11 participants with normal or corrected-

to-normal vision (contacts were allowed, but glasses were not because of visual-field occlusion).

Participants provided informed consent and participated for $10 or course credit. Each

participant was tested in three sessions, each on a separate day. Each expression (fear, neutral,

and disgust) was used once per session. For each run, I used the HFA’s central 24-2 full-

threshold test program, which recorded responses to detected light stimuli of varying luminance

at 54 locations, arranged in a 6° grid on an equiluminant, equidistant visual-field hemisphere.

The HFA ensured central fixation throughout. The stimulus was a white light subtending 0.43°

of varying luminance shown for 200 ms on a white bowl surface (10 cd/m2). The HFA reports

sensitivity in decibels: 10 × log [maximum luminance ÷ detection threshold] dB.

39

After providing consent, participants were trained in posing facial expressions (see chapter 2.1.2)

in a dimly lit room (ambient light, 14 lux). They were then adapted for 5 min in a dark room lit

only by the HFA hemisphere. They remained in this room until the session ended. Participants

responded to a detected light by pressing a button with their right hand. Expressions were posed

and held for 20 s at a time, with 6 to 8 s breaks between poses. Pupil size was measured during

the HFA’s automated gaze initialization at the start of each run, while posing the expression for

that run.

Results

Eyes that express fear and disgust (Figure 4.2a) influenced sensitivity throughout the visual field,

most strongly in the periphery (Figure 4.2b). This result is consistent with the altered occlusion

associated with eye opening and closing, shown to alter the visual field in chapter 2 (see also

Susskind et al., 2008) and serving as a manipulation check for correct posing of fear and disgust

expressions. However, for the primary analysis, I examined the data for only the most central

locations (4.2° surrounding the fovea) to eliminate potentially confounding bias from occlusion

effects in the periphery. The central-visual-field sensitivities were averaged for each run and

submitted to a 3 (expression: fear, neutral, disgust) × 3 (session: 1, 2, 3) repeated measures

analysis of variance (ANOVA). As the model predicted, there was a main effect of expression,

F(2, 20) = 6.4, p = .0072, ηp2 = .39. Specifically, sensitivity was greater when participants posed

fear expressions than when they posed neutral expressions, F(1, 10) = 5.8, p = .036, ηp2 = .37, or

disgust expressions, F(1, 10) = 8.3, p = .016, ηp2 = .45 (Figure 4.2d).

These effects, rather than showing a categorical difference between expressions, appeared to

reflect opposing ends of a continuum of structural variance—between which neutral resides—as

illustrated by a strong linear positive correlation across all runs between peripheral-field (20.6° ±

2.1) sensitivity and central sensitivity, r(97) = .739, p < .0001 (Figure 4.2c). That is, as eye

aperture increased (indexed by increased peripheral sensitivity from reduced occlusion),

parafoveal sensitivity increased, as the model predicted.

The HFA reported several experimental control measures, which were analyzed in separate

Expression (3) × Session (3) repeated measures ANOVAs. No pupil-size differences were found

between expressions, F(2, 20) = 0.13, p > .8, which indicates a lack of autonomic feedback and

40

suggests that the effect of expression was primarily optical rather than a secondary effect of

emotional embodiment (e.g., Niedenthal, 2007). Because of occlusion, some gaze initializations

for disgust-expression runs were performed while the participants posed neutral expressions.

However, pupil size did not differ between fear and neutral expressions, F(1, 10) = 1.0, p > .7.

Responses also did not differ with practice, as there was no main effect of session on central

sensitivity, F(2, 20) = 1.2, p > .3. Expression also did not have a significant effect on the number

of fixation losses, F(2, 20) = 0.03, p > .9; false positive responses, F(2, 20) = 2.1, p > .14; or

false negative responses F(2, 20) = 1.5, p > .23. In conjunction, these data suggest that changes

in the ocular adnexa (i.e., the structural envelope around the eye) are primarily responsible for

the altered perceptual sensitivity associated with fear and disgust expressions.

Figure 4.2. Measured perceptual effects of eye aperture. Examples of statistically modeled fear and disgust expressions are shown in (a). Relative sensitivity maps (b) were created by averaging results at each point of the visual field and subtracting sensitivity associated with neutral expressions from sensitivity associated with fear expressions (top) and from sensitivity associated with disgust expressions (bottom). Hotter and cooler colors indicate greater positive and negative differences, respectively, relative to neutral. Fixation was at (0°, 0°). The dotted green circles indicate the centrally measured locations (4.2° visual angle from fovea). The graph in (c) shows central visual-field sensitivity as a function of peripheral visual-field sensitivity (mean visual angle from fovea = 20.6°, SD = 2.1°). Greater peripheral sensitivity is an index of eye opening. The graph in (d) shows mean sensitivity (left y-axis) and acuity (right y-axis) as a function of expression. Sensitivity scores are restricted to the central visual field. Acuity was measured as the number of correctly read rows of eye-chart letters; these scores were collapsed across contrast and diopter. Higher scores indicate greater sensitivity or acuity. Error bars represent s.e.m.

4.3 Experiment 5: Measuring Acuity Method

To test the acuity model, I measured the visual acuity of 26 participants with normal or

corrected-to-normal vision. Participants provided informed consent and participated for $10 or

course credit. Prior to the experiment, participants were trained in posing facial expressions (see

chapter 2.1.2). Participants were then given goggles (magnetic-resonance-compatible

41

prescription glasses; Safevision, LLC) that myopically blurred their vision. The goggles were of

different optical strengths (+1.0, +2.0, and +3.0 diopters). Given the additive nature of refractive

lens power, participants wore the goggles atop their usual correction glasses or contacts, if any.

Participants were then tested on a set of Bailey-Lovie eye charts (Bailey & Lovie, 1976), nine

with high contrast (black letters) and nine with low contrast (gray letters; 15% Michelson

contrast).

The experiment was divided into blocks by contrast and then by diopter, and the block order was

counterbalanced across participants. Expression order was counterbalanced within participants.

Eye-chart order was randomized across participants. To help the participants stay on task, they

were instructed to make the trained expressions to aid in reading the letters. Participants took

breaks (i.e., they looked away from the chart) between holding expressions as needed. Acuity

was measured as the number of rows of letters read.

Results

Two participants with outlying scores (> 2.5 SD from the mean) were removed from analysis,

which left a final sample of 24. Eye chart acuity scores were submitted to a 3 (expression: fear,

neutral, disgust) × 3 (diopter: +1.0, +2.0, +3.0) × 2 (contrast: high, low) repeated measures

ANOVA. As predicted, there was a main effect of expression, F(2, 46) = 4.7, p = .014, ηp2 = .17

(Figure 4.2c). This effect was modified by an Expression × Diopter interaction, F(4, 92) = 3.0, p

= .022, ηp2 = .12, which revealed a more pronounced effect of expression with increasing need

for acuity to overcome optical aberrations. Analysis of data for the higher diopters (+2.0 and

+3.0) in a 3 × 2 × 2 repeated measures ANOVA revealed a larger effect of expression than in the

previous analysis, F(2, 46) = 7.8, p = .0012, ηp2 = .25, with participants exhibiting greater acuity

when they posed disgust expressions than when they posed neutral expressions, F(1, 23) = 6.9, p

= .015, ηp2 = .23, or fear expressions, F(1, 23) = 11.9, p = .0021, ηp

2 = .34.

Sensitivity-Acuity Trade-off

Finally, I directly analyzed the trade-off between sensitivity and acuity by normalizing and

submitting sensitivity and acuity scores to a 3 (expression: fear, neutral, disgust) × 2

(experiment: sensitivity, acuity) mixed-model ANOVA. As predicted by the optical model

42

(Figure 4.1c), there was a strong crossover interaction, F(2, 66) = 13.7, p < .0001, ηp2 = .29, and

sensitivity increased from disgust to neutral to fear expressions as acuity conversely increased

from fear to neutral to disgust expressions, F(1, 33) = 18.8, p = .0001, ηp2 = .36.

4.4 Chapter Discussion Increasing evidence suggests that emotions influence the central nervous system at multiple

levels to alter visual perception (e.g., Krusemark & Li, 2011; Sherman et al., 2012; Todd, Talmi,

Schmitz, Susskind, & Anderson, 2012). Here I showed that emotional expressions can exert

potent effects at the earliest stage of visual encoding by changing the eyes’ capacity to gather and

focus light. Specifically, expressive eye widening and narrowing influenced the expresser’s

visual perception by altering the eye’s optics—increasing or reducing the exposure of the

refractive cornea. In the context of my thesis, this evidence of early visual filtering supports

Darwin’s first principle of expressive function and his second principle that expressions can be

understood as originating from opposing actions that support opposing functions (Darwin, 1872;

Susskind et al., 2008).

Consistent with the distinct processing dynamics proposed for fear and disgust (Anderson et al.,

2003), as well as their opposing effects on the autonomic nervous system (Levenson, 1992), the

opposing perceptual effects revealed in this chapter shed light on why these two negatively

valenced and avoidance-action-related emotions are associated with opposing facial actions

(Susskind et al., 2008). The expressive widening and narrowing of eye features may converge

with the sympathetic dilation and parasympathetic constriction of the pupil (Beatty & Lucero-

Wagoner, 2000; see also Brunton, 1938, regarding the sympathetically innervated Müller’s

muscle that further opens the eyes), potentially acting as the initial filters toward the

magnocellular (dorsal) and parvocellular (ventral) visual streams (Ungerleider & Mishkin, 1982).

The selective enhancement of sensitivity or acuity, one at the expense of the other, suggests the

differential need for “where” (magnocellular) and “what” (parvocellular) information. This

selective enhancement is consistent with the distinct theorized functions of fear in promoting

vigilance toward localizing an unknown, potentially moving threat (Öhman & Mineka, 2001;

Whalen, 1998) and disgust in promoting discrimination (Sherman et al., 2012) of different kinds

of threat, such as contaminated foods or disease vectors (Chapman & Anderson, 2012; Rozin,

43

Haidt, & McCauley, 2000). Social exaptation of the functional relationship between fear

expressions and the magnocellular system is evidenced by their perceptual prioritization (West,

Anderson, Bedwell, & Pratt, 2010) and low-spatial-frequency tuning (Vuilleumier, Armony,

Driver, & Dolan, 2003). In contrast, I predict that encoding of disgust expressions would be

slower and more attentionally gated (Anderson et al., 2003), dependent on the ventral

parvocellular system and high-spatial-frequency analysis.

In chapter 2, I noted that disgust’s eye narrowing only seemed to hinder the visual field benefits

of the expresser, perhaps useful as a protective closing off the senses (Susskind et al., 2008). The

effects of eye narrowing shown in this chapter, however, suggest not only a protective function

(which would be best supplied by complete eye closure, rather than just eye narrowing) but a

more useful, discriminative purpose of enhanced visual acuity.

Tempering these findings are some considerations of ecological validity. First, from a functional

standpoint, whereas the standardized optometric tests I used provide experimental precision,

their capacity to allow us to infer utility in the case of real-world scenes is limited. When

considering these expressions’ modern utility, it is worth remembering that electricity and

ophthalmology have essentially solved the problems of darkness and impaired acuity—and these

solutions were not available until recently in human history.

Second, from a social-constructivist perspective (Barrett, 2006a; 2006b), widening and

narrowing of the eyes may not universally characterize fear and disgust expressions,

respectively, especially given the powerful influences of culture (Jack et al., 2012) and social and

body context (Aviezer et al., 2008; Aviezer, Trope & Todorov, 2012) on perception of facial

expressions. However, if fear and disgust expressions were swapped, they would serve equally

well as social signals of mental states but would have misaligned functional consequences (e.g.,

reducing acuity in disgust, as shown here, or making it harder to tell where someone is gazing

during fear; see chapter 3). Thus, I argue that these functional benefits probably served as

anchoring sources of invariance in expression perception across cultures and contexts. This

argument is supported by the fact that powerful contextual effects do not cause narrow-eyed

expressions to be judged similarly to wide-eyed ones, and vice versa (Aviezer et al., 2008).

The effects of eye widening and narrowing seen here are tied to a continuum of physical

reconfigurations of eye aperture (Figure 4.2c) rather than to discrete facial configurations,

44

emphasizing an underpinning of a physical nature, rather than psychological categories. The

physical underpinning of these optical effects, which can occur in the absence of their discrete

emotions such as fear and disgust and their associated autonomic expression, suggest that the

egocentric functional dimension of eye opening may extend to other expressions (e.g., raising

eyebrows in surprise or lowering them in anger; Susskind & Anderson, 2008).Therefore, such

effects may provide a window (beyond basic emotions) into the intentions of the expresser and

an optical basis for the ability to communicate complex mental states from the eyes (Baron-

Cohen et al., 2001). In the following chapter, I test precisely that. By extending these egocentric

effects allocentrically, I examine whether there is an optical basis for the ability to read complex

mental states from the eyes.

If our expressions were arbitrary configurations, they would show little cross-cultural

correspondence (Ekman, 1999; Ekman, Sorenson, & Friesen, 1969; Izard, 1994). But rather than

being a collection of discrete, independent categories (Ekman, 1999; Ekman, Sorenson, &

Friesen, 1969), our expressions likely adhere to some underlying universal functional principles

(Darwin, 1872; Susskind et al., 2008) reflected in modern dimensional approaches to facial

expressions and their meaning (Oosterhof & Todorov, 2008; Russell & Barrett, 1999). I show

here that one such potential dimension of expressive invariance across cultures and contexts is

rooted in opposing facial muscle actions around the eyes that arose to harness invariant

principles of light.

45

5 Reading what the mind thinks from how the eye sees “The eyes are the windows to the soul.” Cicero's oft-quoted adage is not merely poetic hyperbole

but captures our remarkable ability to read highly complex mental states from the eyes alone

(Baron-Cohen et al., 2001). The additional contrast afforded by the human white sclera, unique

among primates (Kobayashi & Kohshima, 1997), highlights how our eyes have evolved to

support their salient role in human social and emotional communication. Indeed, the superior

temporal sulcus and gyrus contain neurons responsive not only to the eyes (Allison, Puce,

McCarthy, 2000; Calder et al., 2007) but also neighbor regions that support how we represent the

minds of others (Saxe & Powell, 2006). It remains unknown, however, which specific features of

the eyes and surrounding tissue convey such complex states, and how that came to be.

In chapter 4, I showed that eye widening in fear versus eye narrowing in disgust exerts opposing

optical consequences on how we see in order to serve each emotion’s theorized function.

Operating by the same physical principles of a camera lens aperture, eye widening enhanced

sensitivity, gathering more light information for fear’s vigilance function (Öhman & Mineka,

2001; Whalen, 1998), while eye narrowing enhanced acuity, exerting sharper focus of light

information for disgust’s discriminative function (Chapman & Anderson, 2012; Sherman, Haidt,

& Clore, 2012). Here I investigated whether these opposing eye features shaped for their optical

function explain how we read complex mental states—that is, whether the widening versus

narrowing of eye aperture has been socially co-opted to denote opposing mental states of

information gathering sensitivity versus information discrimination.

I grounded my investigation in the six basic emotional expressions (anger, disgust, fear, joy, sad,

and surprise; Ekman, 1999; Ekman, Sorenson, & Friesen, 1969), the facial features of which

have been shown to communicate more than just basic emotions, such as complex personality

traits (Said, Sebe, & Todorov, 2009). I thus modeled the eyes and surrounding features of the six

basic emotions, creating an exemplar for each basic expression by averaging across affective

faces from two databases (Ekman & Friesen, 1976; Matsumoto & Ekman, 1988). In Experiment

6, I collected participant ratings of 50 different mental states to these exemplars. I first examined

whether the eye region alone can accurately convey the six basic emotions. I then took a

dimensional approach (Oosterhof & Todorov, 2008; Russell & Barrett, 1999; Susskind et al.,

2008) to understanding how we read what another mind thinks from the eyes by mapping the

46

many-to-many relationship between ratings of all 50 mental states and the multidimensional eye

features of the exemplars (Figure 5.1). Aligned with my thesis that perceived mental states are

rooted in to features shaped for sensory function, I hypothesized that mental states conveying

information sensitivity would group along eye widening features while those conveying

information discrimination would group along eye narrowing features, and that these mental

state groups would oppose one another. Then, in Experiment 7, I tested the importance of the

eyes in communicating these same mental states, predicting that their perception would be

maintained even in the context of incongruent information from the remainder of the face.

5.1 General Methods: Expression Modeling I created 6 expression exemplars (Figure 5.1) using a statistical appearance model (Cootes,

Edwards, & Taylor, 2001; Susskind et al., 2008), averaging anger, disgust, fear, joy, sad, and

surprise expressions across two facial affect databases (Ekman & Friesen, 1976; Matsumoto &

Ekman, 1988) then scaling by 1.5 to mitigate averaging’s dampening effects. A reference

exemplar was also created (average of averaged exemplars). All exemplars were aligned at the

irises and cropped around the eye region using Adobe Photoshop and equated for low-level

properties (histogram & spectra match) using SHINE (Willenbockel et al., 2010).

For multidimensional feature analysis, I extracted 7 unique eye features from each exemplar (eye

aperture, eyebrow: distance, slope, and curvature, and wrinkles: nasal, temporal, and below the

eyes). Euclidean coordinates from the averaged appearance models were used to compute eye

aperture (vertical distance from top to bottom of eye), eyebrow distance (vertical distance from

top of eye to intersection with eyebrow), eyebrow slope (from start to end), and eyebrow

curvature (angular change from start to end). Wrinkle features were computed as the amount of

high spatial frequency information extracted from matching rectangular image areas nasal,

temporal, and below the eye—areas were Fourier transformed, high pass filtered using

predefined filter specifications (Vuilleumier, Armony, Driver, & Dolan, 2003), then their

amplitude spectra were radially summed and integrated across frequencies (performed using

Matlab 7.6). All features, except nasal wrinkle, were averaged bilaterally. Finally, each feature

was normalized across exemplars for a final set of 7 features × 6 exemplars.

47

5.2 Experiment 6: Mental states map To examine the relationship between eye features and a variety of mental states, I collected

participant ratings of the 7 exemplar eyes (6 basic categories plus 1 reference exemplar) on

degrees of 50 different mental state terms (6 basic emotions and 44 complex mental states that

expanded on Plutchik’s 1980 set) then analyzed the ratings’ multivariate relationship to the 7 eye

features.

Method

Twenty-eight participants provided informed consent and participated for course credit.

Participants rated the eyes in a within, randomized, full factorial design 7 exemplars × 50 mental

states × 2 repetition blocks (repetitions were averaged). Each trial showed a fixation cross (500

ms), then a target screen of an eye exemplar (subtending 7.5 × 4.7°) with a mental state term and

rating scale below (1 to 9 indicated by “Not at all” to “Very strongly”, respectively). The target

screen remained on for 8000 ms or until response. A break, up to 1 minute, was provided

halfway. The experiment was run on a PC running E-Prime 1.1.

To map mental content onto eye features, I computed independent correlations for each 7

features × 50 mental states combination, creating a matrix of r-values treated as coordinates in a

7-dimension feature space for each of the 50 mental state terms. I then computed the

dissimilarity (Euclidean distance) between all mental state terms such that two terms similarly

correlated across features (e.g., +r, +r; –r, –r) would be closer together while two terms inversely

correlated (e.g., +r, –r) would be further apart. These dissimilarities were visualized using

circular unidimensional scaling (Hubert, Arabie, & Meulman, 2006) (distance variance

accounted for: 54.0%) to chart a navigable map of mental states based on features (Figure 5.1).

Results

I first examined whether mental states associated with basic emotions can be read from the eye

region alone (Baron-Cohen, Wheelwright, & Jollife, 1997). Pairwise basic emotion comparison

accuracy was computed within each participant (e.g., fear eyes accuracy was the percentage of

times fear rating was greater than the remaining 5 basic emotions, with ties counting as chance,

50%). Overall accuracy was very high: 90.4% (one-sample t-test: t(27) = 20.0, p < .0001; 50%

48

chance), ranging from 84.6% for anger (t(27) = 7.9, p < .0001) to 96.4% for surprise (t(27) =

19.6, p < .0001). This was comparable to the basic emotion information conveyed by full

expressions: 90.2% (t(28) = 25.2, p < .0001; see Experiment 7 for details). An extended

comparison of the eye region’s ability to convey basic emotions (i.e., compared to all remaining

49 mental states) produced similar results: overall accuracy, 88.8% (t(27) = 24.2, p < .0001).

Confirming the importance of the eye-opening dimension for mental state content, not only were

the structural features judged highly similar for eye widening fear and surprise, and eye

narrowing disgust and anger (Susskind & Anderson, 2008; Susskind et al., 2008), these pairings

opposed one another as highly dissimilar (Figure 5.1). Largely orthogonal to this opposition, eye

features of joy and sadness were also judged to represent highly dissimilar states.

A principal components analysis (PCA) of the full feature space confirmed these as the primary

and secondary dimensions. The primary dimension was represented by eye widening features:

eye aperture (+0.86; range: +1.0 to –1.0), eyebrow distance (+0.98), and eye narrowing features

such as nasal wrinkles (–0.93), wrinkles below the eyes (–0.92), and eyebrow slope (–0.87). The

secondary dimension was represented by valence-related features: temporal wrinkles (+0.92)

(maximal in the joy exemplar, i.e., “Duchenne eyes”; Duchenne, 1862) and eyebrow curvature (–

0.72) (maximally opposed between the sad and joy exemplars). Together, these two dimensions

captured 88.8% of the total variance; 61.7% by the primary, widening-narrowing dimension

alone. Confirming that these dimensions were not just driven by the basic emotions, a PCA of

the same feature space with the basic emotions removed still captured 88.5% variance together

and 58.4% by the widening-narrowing dimension alone.

Examining the complete map of mental states based on eye features revealed that the eye

narrowing and associated enhanced perceptual discrimination of disgust aligned with a cluster of

mental states that convey social discrimination, such as hate, suspicion, aggressiveness, and

contempt. Opposing these mental state attributions, eye widening and associated enhanced

perceptual sensitivity of fear was associated with information sensitivity, aligned with awe,

anticipation, cowardice, and interest. A rank-order of distances in this multidimensional space

revealed that awe and hate were the most opposing complex mental states communicated

through eye features. Individual subject analyses further revealed that awe and hate were the

most opposing mental states across all participants, significantly different from the average of all

49

946 word-pair distances (mean distance score of 1.15, t(27) = 8.32, p < .0001, where mean

distance score across all [44 × 43 / 2 = 946] word pairs = 0).

Figure 5.1. Relationship between mental states based on eye features. Mental states similar across features appear closer together. Basic emotion states matching the exemplar eye stimuli have been colored for reference. The opposition of disgust and anger (eye-narrowing enhancing discrimination) to fear and surprise (eye widening enhancing sensitivity) is illustrated in their maximal distance around the circle. Orthogonal to the sensitivity-discrimination opposition is a valence-related appetitive-aversive opposition anchored by joy and sadness.

5.3 Experiment 7: Reading eyes in mixed expressions The above results reflect judgments of eyes in isolation. I next investigated the importance of the

eyes’ contributions toward evaluating the mental contents of more complex, full expressions. If

eye widening versus narrowing are diagnostic signals of internal mental states, then observers

should use them even in the context of conflicting expression features. To test this, I created a set

of 49 chimeric expressions (see Figure 5.2a and Figure A2 for examples), seamlessly combining

the 7 upper (“eyes”) and 7 lower (“mouth”) regions of the expression exemplars modeled above

using Adobe Photoshop. I then selected a subset of appropriate mental state terms that uniformly

50

covered the mental state map (Figure 5.1): 6 basic states (disgust, anger, fear, surprise, joy,

sadness), 6 complementary complex states (hate, suspicion, awe, cowardice, admiration,

apprehension), and 4 in between (interest, boredom, pride, and remorse).

Method

Twenty-nine new participants provided informed consent and participated for course credit.

Participants rated all 49 faces on all 16 mental state terms in a within, randomized, full factorial

design (49 × 16 trials). Participants rated the entire expression (no mention of the eyes were

made). Faces subtended 7.5 × 7.5°, and experimental setup and trial structure were identical to

the previous experiment.

Results

As in Experiment 6, I first examined the accuracy of basic emotions conveyed by the 6 basic full

expressions (i.e., matching eye and mouth regions). Pairwise basic emotion comparison accuracy

was computed similarly using the 6 basic emotion ratings, and overall performance was similarly

very high: 90.2% (one-sample t-test: t(28) = 25.2, p < .0001; 50% chance), ranging from 82.8%

for fear (t(28) = 8.9, p < .0001) to 94.5% for surprise (t(28) = 20.3, p < .0001).

I then analyzed all faces excluding the reference average exemplar (36 total). From this dataset

(6 eyes × 6 mouth × 16 mental states × 29 participants), I examined how the eyes of expressions

differentiated mental state perception by averaging across the mouth dimension. Each

participant’s remaining 6 × 16 data matrix was treated as coordinates in 16-dimension mental

state space, and I computed the dissimilarities (Euclidean distance) between each of the 6 basic

emotion eyes. Then, using linear unidimensional scaling (Hubert, Arabie, & Meulman, 2006),

these relationships were visualized along a single dimension (Figure 5.2b). Mental state

perceptions of narrow eyes of disgust and anger strongly opposed those of wide eyes of fear and

surprise, even when in the context of competing full face expression features.

Next, I conducted univariate analyses of specific mental state perception toward “narrow” versus

“wide” eyes. From the same data matrix as above (collapsed across the mouth dimension), I

averaged disgust and anger eyes for “narrow” scores and averaged fear and surprise eyes for

“wide” scores. First, examining basic emotions perception, paired-sample t-tests confirmed that

51

narrow versus wide eyes differentiated perceptions of disgust and anger (t(28) = 12.0, p <

.0001), fear and surprise (t(28) = –11.5, p < .0001), and in opposing directions (2 × 2 repeated

measures ANOVA interaction, F(1, 28) = 166.9, p < .0001; Figure 5.2c). Critically, similar

analyses examining perception of complex mental states predicted by the mental state map

revealed that narrow versus wide eyes differentiated hate and suspicion (t(28) = 11.4, p < .0001),

awe and cowardice (t(28) = –8.2, p < .0001), and in opposing directions (2 × 2 repeated

measures ANOVA interaction, F(1, 28) = 137.7, p < .0001; Figure 5.2c). Thus eye widening

versus narrowing were used as powerful cues for these opposing mental state attributions, even

in the context of potentially incongruous information from the rest of the face.

Lastly, I examined the diagnosticity of the eye features along the valence-related dimension in

the context of incongruent facial information. I computed the pairwise comparison accuracy of

joy and sad eyes with non-matching mouths. Sad eyes were diagnostic of sadness, compared to

other basic emotions, 74.4% (t(28) = 9.0, p < .0001; this was comparable to other eyes, which

ranged from 70.8% to 75.2% accuracy, Ps < .0001). However, joy eyes were perceived as joy

only with 23.0% accuracy, significantly less than 50% chance (t(28) = –12.1, p < .0001). Paired-

sample t-tests showed that narrower joy eyes, in the context of incongruent mouth information

(i.e., not smiles), were reliably confused as discriminatory mental states, with significantly

greater ratings for disgust and anger compared to joy (t(28) = 7.5, p < .0001; likewise for hate

and suspicion compared to joy, t(28) = 8.7, p < .0001), but no difference in ratings of fear and

surprise compared to joy (t(28) = 0.3, p > .75; likewise for awe and cowardice compared to joy,

t(28) = 1.2, p > .23).

52

Figure 5.2. Effect of eyes on mental state perception in full and mixed expressions. Exemplars of disgust eyes (left group) and fear eyes (right group) with mixed mouths (disgust, anger, surprise, and fear; left to right) are shown in (a). Relationship between exemplar eyes of full expressions, across all mouth combinations are shown in (b). The opposition of narrow disgust and anger eyes to wide fear and surprise eyes is illustrated in their maximal distance apart. Distance locations shown are mean ± s.e.m. Examining specific mental states ratings in (c) showed that narrow versus wide eyes strongly differentiated their matching basic (left) and complex mental states (right), as predicted by the mental state map in Figure 5.1. Data points are mean ± s.e.m.

5.4 Chapter Discussion In this chapter, I showed how opposing facial features around the eyes that arose to harness

invariant properties of light (chapter 4) were socially co-opted to convey our complex inner

states, serving as windows into our mental landscapes.

I first found that the eyes alone conveyed the six basic emotions with high accuracy (90.4%),

comparable to full expressions (90.2%). However, the same eye features conveyed much more

complex information than just the basic emotions, as shown previously (Baron-Cohen et al.,

2001; Said, Sebe, & Todorov, 2009). Toward capturing the variety of complex mental states that

the eyes can convey, I employed multiple, continuous feature dimensions, along which basic

expressions’ features were used as points of anchor. This dimensional perspective was supported

by my previous findings that the opposing optical effects of fear versus disgust eye opening were

53

not discrete but observed along an eye-opening continuum (chapter 4). This multidimensional

analysis revealed that the eye widening versus narrowing features associated with perceptual

enhancements of sensitivity versus discrimination for the expresser likewise supported how the

eyes convey complex mental states of information sensitivity versus discrimination, such as awe

and suspicion. The primacy of eye opening features was further emphasized in how these

opposing mental states were communicated even in the context of incongruent lower facial

information.

Orthogonal to the primary sensitivity-discrimination dimension, I found in our mental state map

(Figure 5.1) an appetitive-aversive dimension anchored by joy and sadness, indicating that a

sensitivity-discrimination antagonism is not the only principle of how the eyes communicate

internal states. In this dimension, I found that eye opening features were less diagnostic of

mental states. But interestingly, narrower joy eyes, without their contextual smiles, were

perceived as discriminative mental states associated with eye narrowing in the primary

dimension (see Mattson et al., 2013, for infant evidence showing that the temporal wrinkles, or

“Duchenne eyes”, Duchenne, 1862, can convey intensity across opposing valence contexts).

The primacy of the sensory dimension, over the valence dimension, underscores Darwin’s (1872)

theories on expressive features’ sensory origins. However, the features’ multidimensionality is

important to note here, and a broader exploration of the mental states map integrating the

primary and secondary dimensions revealed interesting potential accounts of other mental states.

For instance, pride and boastful opposed remorse and submission, with the latter between fear

and sadness, and the former between disgust and joy. Mental states of pride may originate from

an appetitive form of discrimination, while remorse and submission stem from an aversive form

of sensitivity. Similarly, boredom (between sadness and disgust) opposed interest (between joy

and surprise). Interest may originate in an appetitive sensitivity and boredom from an aversive

scrutiny.

Another salient aspect of the mental state map is its asymmetry. It is possible that the relative

abundance of mental states on one side may reflect an uninteresting sampling bias of states

toward anger or disgust. Alternatively, a true asymmetry may reflect the physical signal bias

revolving around our unique eye whites (Kobayashi & Kohshima, 1997). Revealing more sclera

in eye widening has demonstrated to enhance processing of the eyes (chapter 3; see also Adolphs

54

et al., 2005; Whalen et al., 2004). This increase in low-level contrast and luminance (among

these exemplars, from lowest disgust to highest fear, there were increases of 3.1 s.d. in contrast

and 2.7 s.d. in luminance) sends a physically stronger signal, which may bias signal detection

and diagnosticity of the possible mental states attributable to it. Conversely, eye narrowing, by

shifting the salience away from eye whites to other features surrounding the eyes requires finer

discrimination of higher frequency information (e.g., wrinkles) across a wider permutation of

possible configurations. Thus the opposing eye features which originated for enhancing visual

sensitivity versus discrimination (chapter 4) may be mirrored in how the receiver needs to

decode them. While wide eyes may represent a relatively unambiguous signal and associated

mental state attributions, narrow eyes require greater discrimination to differentiate among

underlying mental states.

It is worth noting that the eye features I used here are not the only variables that influence mental

state attribution. For example, my examination held constant directional eye gaze, which can

influence emotional expression perception (Adams & Kleck, 2005) and is an important

contributor to mental state decoding (Baron-Cohen, 1995). Evidence also suggests that eye

features have been co-opted to reflect more complex social pressures, such as their widening

versus narrowing that make appearances more juvenile versus mature (Marsh, Adams, & Kleck,

2005; Zebrowitz, 1997), potentially explaining why similarly negative expressions, such as fear

and anger, solicit asymmetric approach versus avoidance responses from perceivers (Marsh,

Ambady, & Kleck, 2005).

In sum, these results suggest the conspicuous facial features that evolved to harness physical

principles of light may explain how we convey the eye-widening experience of awe or the

narrow-eyed mindedness of hate. Thus the expressiveness that alter how the eye sees reveals an

important organizing principle of our ability to read the complex states of another’s mind,

supporting the theory that our expressive features that evolved for function were co-opted for

social communication.

55

6 Summary and Conclusions The purpose of my dissertation was to bridge a gap in our understanding of facial expressions:

why they look the way do and how they were shaped to be the social communicative signals of

today. My thesis is that our facial expressions originated for sensory function, providing

egocentric benefit to the expresser, then these adaptive features were socially co-opted for

allocentric function to the expressions’ observers. I grounded my thesis in Darwin’s (1872)

principles of facial expressions, which theorizes their origins in expresser’s function and as

organized along a dimension of sensory opposition (Susskind et al., 2008). In order to test my

thesis, I focused on the eyes, showing how the facial expressive features surrounding the eyes

alter how we take in sensory information in a situationally appropriate manner. I then showed

how the eye features associated with adaptive sensory processing also serve a social function

through the altered information transmitted to our neighbours.

Summary of Studies

In chapter 2, I showed evidence for Darwin’s (1872) first principle of facial expressions’ sensory

function. Specifically, wider eye opening in fear expressions conferred an egocentric benefit by

enhancing the visual field 9.4% farther out in the available visual periphery of the expresser.

This enhancement is congruent with fear’s theorized function of vigilance (Öhman & Mineka,

2001; Whalen, 1998), increasingly the likelihood of the detection and location of potential

threats.

In chapter 3, I showed evidence for how the same eye widening features of fear were co-opted

socially, conferring an allocentric benefit to expressions’ receivers by way of an enhanced

physical gaze signal. Directionality of wider eye gazes of were perceived more accurately and

facilitated faster responses to locating eccentric targets. This benefit was neither driven by the

perceived emotion nor attention, but rather an enhanced physical signal that originated from

greater exposure of the iris and the physical salience of our uniquely white sclera (Kobayashi &

Kohshima, 1997). Thus, the functional essence of expressive fear at its basic sensory level (in

locating potential threat) is passed on to the observer through transmission of a clearer “look

here” gaze signal. These results highlight the co-evolution of sensory and social functions of

56

emotional expressions, with eye widening serving to enhance processing of important

environmental events in the visual fields of both expresser and observer.

In chapter 4, I showed evidence for Darwin’s (1872) second principle of facial expressions’

sensory function, as conferring egocentric benefit to the expresser along a continuous dimension

of an eye widening versus narrowing opposition. Specifically, I showed how opposing

expressive features that widen versus narrow the eyes harness a basic principle of light to serve

as a functional trade-off between perceptual sensitivity and discrimination. Facial features along

a dimension of eye opening that expose or conceal the cornea, which accounts for two-thirds of

our eye’s refractive power (Duke-Elder & Abrams, 1970), enhanced the gathering of light in eye

widening fear expressions or the focusing of light in eye narrowing disgust expressions. While

these opposing sensory effects served the opposing theorized functions of fear and disgust

(Chapman & Anderson, 2012; Öhman & Mineka, 2001; Rozin, Haidt, & McCauley, 2000;

Whalen, 1998), they also demonstrated continuous rather than categorical effects of visual

perception, supporting a physical dimension of expressive function.

In chapter 5, I showed that these conspicuous features that widen versus narrow the eye aperture,

associated with perceptual sensitivity versus discrimination for the expresser, similarly conveyed

basic (e.g., fear versus disgust) and complex mental states (e.g., awe versus suspicion) of

sensitivity versus discrimination. Features along the eye opening dimension accounted for the

majority of variance in mental state attribution (61.7%), and maintained their social signaling

even in the context of incongruent facial information. Further resonating the sender’s perceptual

functions for the expression’s receiver, sensitivity-enhancing wide eyes were attributed fewer

mental states (for greater diagnosticity) compared to discrimination-enhancing narrow eyes were

attributed more mental states (requiring greater discrimination) (Figure 5.1). These results

suggest that the adaptive origins of how the eye sees have shaped the external reflection of

internal mental states and the human capacity to read them.

Uniting Theoretical Views

Taken together, the evidence presented here shows how facial expressive features that convey

social information originated in adaptive purpose for the expresser. But beyond that, the

evidence for this link potentially offers a resolution of the two major theoretical divisions in

facial expression research, the categorical view and the constructivist/dimensional view.

57

The studies here utilized the expressive features of fear and disgust, testing the sensory effects

against the emotions’ theorized functions. Thus, in its initial framing, my thesis adopted the basic

emotional expressions posited by the categorical view (Ekman, 1999, Izard, 1994). However, it

differs from the categorical view in that it adheres chiefly to Darwin’s (1872) theoretical

principles. Thus “fear” and “disgust” expressions are considered not as universal categories but

as action tendencies predicated on function and organized as opposites along a dimension of an

expressive continuum (Oosterhof & Todorov, 2008; Russell & Barrett, 1999; Susskind et al.,

2008).

One consequence of this perspective, different from the categorical view, is the availability of a

wide variety of facial expressions interpretable as signals of different mental states (Baron-

Cohen, Wheelwright, & Jollife, 1997; Baron-Cohen et al., 2001; Du, Tao, & Martinez, 2014).

Meanwhile, the evidence for expressions’ functional basis shown here provides an equally

parsimonious, empirical account of the cultural consistency of basic expressions (Ekman,

Sorenson, & Friesen, 1969). This perspective also accommodates the constructivist view in that

the labels that define specific facial actions and their degrees of expressivity (i.e., what the

expressions are) is left up to social interpretation and context (e.g., Aviezer, Trope, & Todorov,

2012; Fridlund, 1997; Jack et al., 2012; Russell & Barrett, 1999).

But perhaps the most important consequence of the functional dimensional perspective of my

thesis is that it provides guiding, not rigid, constraints for understanding why our expressions as

social signals look the way they do. While features could be arbitrarily mapped for

communication, they cannot be arbitrarily mapped for function (Darwin, 1872; Susskind et al.,

2008). Then, rather than the emergence of completely new and variant sets of expressive forms

across different cultures (Ekman, Sorenson, & Friesen, 1969), it is more likely that these

adaptive action tendencies were socially co-opted, serving as anchoring sources of invariance in

expression perception across cultures and contexts (Andrew, 1963; Shariff & Tracy, 2011). For

example, from a strictly constructivist perspective (Barrett, 2006a; 2006b), widening and

narrowing of the eyes may not universally characterize fear and disgust expressions,

respectively, especially given the powerful influences of culture (Jack et al., 2012) and social

context (Aviezer et al., 2008; Aviezer, Trope, & Todorov, 2012) on perception of facial

expressions. However, if fear and disgust expressions were swapped, they would serve equally

well as social signals of mental states but would have misaligned functional consequences (e.g.,

58

reducing acuity in disgust, see chapter 4; or making it harder to tell where someone is gazing

during fear, see chapter 3). Evidence that these functionally-driven features serve as guides for

social interpretation is supported by the fact that powerful contextual effects do not cause

narrow-eyed expressions to be judged similarly to wide-eyed ones, and vice versa (Aviezer et al.,

2008).

This perspective may also serve as a useful foundation for navigating the highly complex ways

that facial information can interact with our social nature. For instance, neuroimaging evidence

has shown that amygdala activations toward fear expressions are enhanced for members one’s

own culture, suggesting greater saliency attributed to potential threats toward in-group members

(Chiao et al., 2008). Also, the pedomorphic features of fear’s wide eyes have been shown to

solicit more socially sympathetic responses from neighbors (Marsh, Adams, & Kleck, 2005),

thereby reciprocating the signaling function transmitted by the expresser (chapter 3). The

overlearning of these socially co-opted expressive facial features (Zebrowitz, 1997) can also

convey traits that are more enduring than transient mental states, such as sexual orientation

(Rule, Ambady, Adams, Macrae, 2008) and political affiliation (Rule & Ambady, 2010). And

these overlearned features combined with how our social nature feeds back on itself may explain

the self-fulfilling prophecies of how faces that communicate competence can predict election

outcomes (Todorov, Mandisodza, Goren, & Hall, 2005) and financial measures of success (Rule

& Ambady, 2008).

Converging Evidence

Multiple lines of evidence provide converging support for the functional basis of expressions and

their social exaptation. Beginning with the expresser, the widening versus narrowing of eye

features may converge with their ties to the autonomic system, which is associated with

sympathetic tone in fear versus parasympathetic tone in disgust (de Jong, van Overveld, &

Peters, 2011; Levenson, 1992) and coincides with the sympathetic dilation versus

parasympathetic constriction of the pupil (Beatty & Lucero-Wagoner, 2000). The muscular

anatomy of our eyes converges toward a similar autonomic function, as seen in the

sympathetically innervated Müller’s muscle (Brunton, 1938) that would further open the eyes

with sympathetic tone in fear. These widening versus narrowing facial actions that filter light

information to a sensitivity versus acuity opposition align with the crude but fast magnocellular

59

versus slow but sharp parvocellular visual systems (Livingstone & Hubel, 1987), which are then

channeled toward the “where” (dorsal) versus “what” (ventral) visual streams (Ungerleider &

Mishkin, 1982).

Towards transmitting these functional benefits of expressers to observers, our physical eye

features may have co-evolved with our social nature. For example, working in conjunction with

eye widening in fear to send conspicuous physical signals to our neighbours is the morphology

of our eyes, elongated for improved directional gaze information and the white sclera to enhance

its physical contrast (Kobayashi & Kohshima, 1997). At this physical level of information

transmission, the asymmetric salience associated with fear’s eye widening was demonstrated in

gaze enhancement in chapter 3 and the diagnosticity of eye widening features in chapter 5

(Figure 5.1). This is likely due to the potential harm associated with fear’s vigilance toward

unknown, moving threats (Öhman & Mineka, 2001; Whalen, 1998), where immediacy is

prioritized over accuracy.

Lastly, further social exaptation of expression function has been demonstrated in the

emotionality of full fear expressions enhancing averted gaze direction processing (Adams &

Franklin, 2009). Fear expressions have also shown to improve early vision for observers (Phelps,

Ling, & Carrasco, 2006), specifically along lower spatial frequency channels (Bocanegra &

Zeelenberg, 2009). This is further aligned with fear expressions’ prioritized perception and

action via the low-spatial-frequency tuned magnocellular pathway, projecting to the dorsal

stream (Vuilleumier, Armony, Driver, & Dolan, 2003; West, Anderson, Bedwell, & Pratt, 2010).

In conclusion, the present and converging evidence support the thesis that our emotional

expressions originated as sensory adaptations for the expresser, and were then co-opted for social

function. Thus, by the same eyes with which we see the world, evolution has provided windows

into each other’s souls. And as we harvest these empirical fruits of Darwin’s insights, we might

find pause in noticing that our emotional eyes not only connect us in the present but across a

natural history of how our once-individual survival was leveraged into a flourishing, co-

operative one.

60

References Adams, R. B. Jr., Gordon, H. L., Baird, A. A., Ambady, N., Kleck, R. E. (2003). Effects of gaze

on amygdala sensitivity to anger and fear faces. Science, 300, 1536.

Adams, R. B., Jr., & Kleck, R. E. (2005). Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion, 5, 3-11.

Adams, R. B., Jr., & Franklin, R. G., Jr. (2009). Influence of emotional expression on the processing of gaze direction. Motivation and Emotion, 33, 106-112.

Adolphs, R, Gosselin, F., Buchanan, T. W., Tranel, D., Schyns, P., & Damasio, A. R. (2005). A mechanism for impaired fear recognition after amygdala damage. Nature, 433, 68-72.

Adolphs, R., Tranel, D., Damasio, H., & Damasio, A. (1994). Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature, 372, 669-72.

Allison, T., Puce, A., & McCarthy, G. (2000). Social perception from visual cues: role of the STS region. Trends in Cognitive Sciences, 4, 267-278.

Anderson, A. K., Christoff, K., Panitz, D. A., De Rosa, E., & Gabrieli, J. D. E. (2003). Neural correlates of the automatic processing of threat facial signals. Journal of Neuroscience, 23, 5627-5633.

Aviezer, H., Hassin, R. R., Ryan, J., Grady, C., Susskind, J. M., Anderson, A. K., . . . Bentin, S. (2008). Angry, disgusted, or afraid? Studies on the malleability of emotion perception. Psychological Science, 19, 724-732.

Aviezer, H., Trope, Y., & Todorov, A. (2012). Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science, 338, 1225-1229.

Bailey, I. L. & Lovie, J. E. (1976). New design principles for visual acuity letter charts. American Journal of Optometry and Physiological Optics, 53, 740-745.

Baron-Cohen, S. (1995). Mindblindness: An Essay on Autism and Theory of Mind. MIT Press/Bradford Books, Boston.

Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The "reading the mind in the eyes" test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry, 42, 241-251.

Baron-Cohen, S., Wheelwright, S., & Jollifee, T. (1997). Is there a “language of the eyes”? Evidence from normal adults, and adults with autism or Asperger syndrome. Visual Cognition, 4, 311-331.

Barrett, L. F. (2006a). Are emotions natural kinds? Perspectives on Psychological Science, 1, 28-58.

Barrett, L. F. (2006b). Solving the emotion paradox: Categorization and the experience of emotion. Personality and Social Psychology Review, 10, 20-46.

61

Beatty, J., & Lucero-Wagoner, B. (2000). The pupillary system. In J. T. Cacioppo, G. Berntson, & L. G. Tassinary (Eds.), Handbook of Psychophysiology (pp. 142-162). Cambridge: Cambridge University Press.

Blake, C., Lai, W., & Edward, D. (2003). Racial and ethnic differences in ocular anatomy. International Ophthalmology Clinics, 43, 9-25.

Bocanegra, B. R. & Zeelenberg, R. (2009). Emotion improves and impairs early vision. Psychological Science, 20, 707-713.

Breiter, H. C., Etcoff, N. L., Whalen, P. J., Kennedy, W. A., Rauch, S. L., Buckner, R. L., Strauss, M. M., Hyman, S. E., & Rosan, B. R. (1996). Response and habituation of the human amygdala during visual presentation of facial expression. Neuron, 17, 875-887.

Bruce, V. & Young, A. (1986). Understanding face recognition. British Journal of Psychology, 77, 305-327.

Brunton, C. E. (1938). Smooth muscle of the periorbita and the mechanism of exophthalmos. British Journal of Ophthalmology. 22, 257-268.

Calder, A. J., Beaver, J. D., Winston, J. S., Dolan, R. J., Jenkins, R., Eger, E., & Henson, R. N. A. (2007). Separate coding of different gaze directions in the superior temporal sulcus and inferior parietal lobule. Current Biology, 17, 20-25.

Calder, A. J., Keane, J., Lawrence, A. D., & Manes, F. (2004). Impaired recognition of anger following damage to the ventral striatum. Brain, 127, 1958-1969.

Calder, A. J., Keane, J., Manes, F., Antoun, N., Young, A. W. (2000). Impaired recognition and experience of disgust following brain injury. Nature Neuroscience, 3, 1077-1078.

Calder, A. J., Lawrence, A. D., & Young, A. W. (2001). Neuropsychology of fear and loathing. Nature Reviews Neuroscience, 2, 352-363.

Chapman, H. A. & Anderson, A. K. (2012). Understanding disgust. Annals of the New York Academy of Sciences: The Year in Cognitive Neuroscience, 1251, 62-76.

Chapman, H. A., Kim, D. A., Susskind, J. M., & Anderson, A. K. (2009). In bad taste: evidence for the oral origins of moral disgust. Science, 323, 1222-1226.

Chiao, J. Y., Iidaka, T., Gordon, H. L., Nogawa, J., Bar, M., Aminoff, E., . . . Ambady, N. (2008). Cultural specificity in amygdala response to fear faces. Journal of Cognitive Neuroscience, 20, 2167-2174.

Collett, D. (1991). Modeling binary data. New York: Chapman & Hall/CRC.

Cootes, T., Edwards, G., & Taylor, C. (2001). Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 681-685.

Cowey, A., & Rolls, E. T. (1974). Human cortical magnification factor and its relation to visual acuity. Experimental Brain Research, 21, 447-454.

Darwin, C. (1872/1998). The Expression of the Emotions in Man and Animals. New York: Oxford University Press.

de Jong, P. J., van Overveld, M., & Peters, M. L. (2011). Sympathetic and parasympathetic responses to a core disgust video clip as a function of disgust propensity and disgust sensitivity. Biological Psychology, 88, 174-179.

62

Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, USA, 111, E1454-E1462.

Duchenne, G. B. (1862/1990). The Mechanism of Human Facial Expression (Cuthbertson, RA Trans.). New York: Cambridge University Press.

Duke-Elder, S., & Abrams, D. (1970). Ophthalmic Optics and Refraction. In S. Duke-Elder (Ed.), System of Ophthalmology (Vol. 5). London: Henry Kimpton.

Ekman, P. (1999). Basic Emotions. In T. Dalgleish, T. Power (Eds.), The Handbook of Cognition and Emotion, (pp. 45-60). John Wiley & Sons, Ltd., Sussex, U.K.

Ekman, P., & Friesen, W. V. (1976). Pictures of Facial Affect. Palo Alto: Consulting Psychologists Press, Palo Alto.

Ekman, P., Friesen, W. V., & Hager, J. C. (1978). Facial Action Coding System. Salt Lake City: Research Nexus.

Ekman, P., Levenson, R. W., & Friesen, W. V. (1983). Autonomic nervous system activity distinguishes among emotions. Science, 221, 1208-1210.

Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164, 86-88.

Etcoff, N. L., & Magee, J. J. (1992). Categorical perception of facial expressions. Cognition, 44, 227-240.

Fox, E., Mathews, A., Calder, A. J., & Yiend, J. (2007). Anxiety and sensitivity to gaze direction in emotionally expressive faces. Emotion, 7, 478-486.

Fridlund, A. J. (1997). The new ethology of human facial expressions. In J. A. Russell & J. Fernandez-Dols (Eds.), The psychology of facial expression (pp. 103-129). Cambridge: Cambridge University Press.

Friesen, C. K., & Kingstone, A. (1998). The eyes have it! Reflexive orienting is triggered by nonpredictive gaze. Psychonomic Bulletin & Review, 5, 490-495.

Fujimura, T., Matsuda, Y., Katahira, K., Okada, M., & Okanoya, K. (2012). Categorical and dimensional perceptions in decoding emotional facial expressions. Cognition & Emotion, 26, 587-601.

Harrison, N., Singer, T., Rotshtein, P., Dolan, R.J., & Critchley, H.D. (2006). Pupillary contagion: central mechanisms engaged in sadness processing. Social Cognitive & Affective Neuroscience, 1, 5-17.

Hubert, L., Arabie, P., & Meulman, J. (2006). The Structural Representation of Proximity Matrices with MATLAB. Philadelphia, Alexandria, VA: ASA-SIAM.

Izard, C. E. (1994). Innate and universal facial expressions: evidence from developmental and cross-cultural research. Psychological Bulletin, 115, 288-299.

Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R., & Schyns, P. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, USA, 109, 7241-7244.

James, W. (1884). What is an Emotion? Mind, 9, 188-205.

63

Jenkins, R., Beaver, J. D., & Calder, A. J. (2006). I thought you were looking at me: direction-specific aftereffects in gaze perception. Psychological Science, 17, 506-513.

Kobayashi, H., & Kohshima, S. (1997). Unique morphology of the human eye. Nature, 387, 767-768.

Krusemark, E., & Li, W. (2011). Do all threats work the same way? Divergent effects of fear and disgust on sensory perception and attention. Journal of Neuroscience, 31, 3429-3434.

Lee, D. H., Mirza, R., Flanagan, J. G., & Anderson, A. K. (2014). Optical origins of opposing facial expression actions. Psychological Science, 25, 745-752.

Lee, D. H., Susskind, J. M., & Anderson, A. K. (2013). Social transmission of the sensory benefits of fear eye-widening. Psychological Science, 24, 957-965.

Langton, S. R. H., Watt, R. J., & Bruce, V. (2000). Do the eyes have it? Cues to the direction of social attention. Trends in Cognitive Sciences, 4, 50-59.

Levenson, R. W. (1992). Autonomic nervous system differences among emotions. Psychological Science, 3, 23-27.

Levenson, R. W., Ekman, P., Heider, K., & Friesen, W. V. (1992). Emotion and autonomic nervous system activity in the Minangkabau of West Sumatra. Journal of Personality and Social Psychology, 62, 972-988.

Lipp, O. V., Price, S. M., & Tellegen, C. S. (2009). No effect of inversion on attentional and affective processing of facial expressions. Emotion, 9, 248–259.

Livingstone, M. S., & Hubel, D. H. (1987). Psychophysical evidence for separate channels for the perception of form, color, movement, and depth. Journal of Neuroscience, 7, 3416-3468.

Marsh, A. A., Adams, R. B., Jr., & Kleck, R. E. (2005). Why do fear and anger look the way they do? Form and social function in facial expressions. Personality and Social Psychology Bulletin, 31, 73-86.

Marsh, A. A., Ambady, N., & Kleck, R. E. (2005). The effects of fear and anger facial expressions on approach- and avoidance- related behaviors. Emotion, 5, 118–124.

Matsumoto, D., & Ekman, P. (1988). Japanese and Caucasian facial expressions of emotion (JACFEE) [Slides]. San Francisco: San Francisco State University, Department of Psychology, Intercultural and Emotion Research Laboratory.

Mattson, W. I., Cohn, J. F., Mahoor, M. H., Gangi. D. N., & Messinger, D. S. (2013). Darwin’s Duchenne: eye Constriction during infant joy and distress. PLOS ONE, 8, e80161.

McKelvie, S. J. (1995). Emotional expression in upside-down faces: Evidence for configurational and componential processing. British Journal of Social Psychology, 34, 325–334.

Morris, J. S., DeGelder, B., Weiskrantz, L., & Dolan, R.J. (2001). Differential extrageniculostriate and amygdala responses to presentation of emotional faces in a cortically blind field. Brain, 124, 1241-1252.

Morris, J. S., Frith, C. D., Perrett, D. I., Rowland, D., Young, A. W., Calder, A. J., & Dolan, R. J. (1996). A differential neutral response in the human amygdala to fearful and happy facial expressions. Nature, 383, 812-815.

64

Morris, J. S., Öhman, A., & Dolan, R. J. (1998). Conscious and unconscious emotional learning in the human amygdala. Nature, 393, 467-470.

Niedenthal, P. M. (2007). Embodying emotion. Science, 316, 1002-1005.

Oosterhof, N. N., & Todorov, A. (2008). The functional basis of face evaluation. Proceedings of the National Academy of Sciences, USA, 105, 11087-11092.

Phelps, E. A., Ling, S., & Carrasco, M. (2006). Emotion facilitates perception and potentiates the perceptual benefits of attention. Psychological Science, 17, 292–299.

Phillips, M. L., Young, A. W., Senior, C., Brammer, M., Andrew, C., Calder, A. J., . . . David, A. S. (1997). A specific neural substrate for perceiving facial expressions of disgust. Nature, 389, 495-498.

Plutchik, R. (1980). Emotion: Theory, Research, and Experience: Vol. 1. Theories of Emotion: Academic Press, New York.

Putman, P., Hermans, E., & van Honk, J. (2006). Anxiety meets fear in perception of dynamic expressive gaze. Emotion, 6, 94-102.

Rolls, E. T. (1990). A theory of emotion, and its application to understanding the neural basis of emotion. Cognition & Emotion, 4, 161-190.

Rovamo, J., Virsu, V., & Näsänen, R. (1978). Cortical magnification factor predicts the photopic contrast sensitivity of peripheral vision. Nature, 271, 54-56.

Rosenstein, D., & Oster, H. (1988). Differential facial responses to four basic tastes in newborns. Child Development, 59, 1555-1568.

Rozin, P., Haidt, J., & McCauley, C. (2000). Disgust. In M. Lewis, J. M. Haviland-Jones (Eds.), Handbook of Emotions (pp. 637-653). New York: Guilford.

Rule, N. O., & Ambady, N. (2008). The face of success: Inferences from Chief Executive Officers’ appearance predict company profits. Psychological Science, 19, 109-111.

Rule, N. O., & Ambady, N. (2010). Democrats and Republicans can be differentiated from their faces. PLOS ONE, 5, e8733.

Rule, N. O., Ambady, N., Adams, R. B., Jr., & Macrae, C. N. (2008). Accuracy and awareness in the perception and categorization of male sexual orientation. Journal of Personality and Social Psychology, 95, 1019-1028.

Russell, B. (1912/1959). The Problems of Philosophy. Oxford: Oxford University Press.

Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178.

Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of cross-cultural studies. Psychological Bulletin, 115, 102– 141.

Russell, J. A., & Barrett, L. F. (1999). Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. Journal of Personality and Social Psychology, 76, 805-819.

Said, C. P., Sebe, N., & Todorov, A. (2009). Structural resemblance to emotional expressions predicts evaluation of emotionally neutral faces. Emotion, 9, 260-264.

65

Sato, W., Kochiyama, T., & Yoshikawa, S. (2011). The inversion effect for neutral and emotional facial expressions on amygdala activity. Brain Research, 1378, 84–90.

Saxe, R., & Powell, L. J. (2006). It's the thought that counts: specific brain regions for one component of theory of mind. Psychological Science, 17, 692-699.

Scherer, K. R., & Wallbott, H. G. (1994). Evidence for universality and cultural variation of differential emotion response patterning. Journal of Personality and Social Psychology, 66, 310-328.

Shariff, A., & Tracy, J. (2011). What are emotion expressions for? Current Directions in Psychological Science, 20, 395-399.

Sherman, G. D., Haidt, J., & Clore, G. L. (2012). The faintest speck of dirt: disgust enhances the detection of impurity. Psychological Science, 23, 1506-1514.

Smith, M. L., Cottrell, G. W., Gosselin, F., & Schyns, P. G. (2005). Transmitting and decoding facial expressions. Psychological Science, 16, 184-189.

Smith, F. W., & Schyns, P. G. (2009). Smile through your fear and sadness: transmitting and identifying facial expression signals over a range of viewing distances. Psychological Science, 20, 1202-1208.

Sprengelmeyer, R., Young, A. W., Calder, A. J., Karnat, A., Lange, H., Hömbert, V., Perrett, D. I., & Rowland, D. (1996). Loss of disgust perception of faces and emotions in Huntington’s disease. Brain, 119, 1647-1665.

Steiner, J. (1973). The gustofacial response: observation on normal and anencephalic newborn infants. In J. F. Bosma (Ed.), Fourth Symposium on Oral Sensation and Perception (pp. 254–278). U.S. Department of Health, Education and Welfare. Bethesda, MD.

Strack, F., Martin, L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: a nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology, 54, 768-777

Susskind, J. M., & Anderson, A. K. (2008). Facial expression form and function. Communicative and Integrative Biology, 1, 148-149.

Susskind, J. M., Lee, D. H., Cusi, A., Feiman, R., Grabski, W., & Anderson, A. K. (2008). Expressing fear enhances sensory acquisition. Nature Neuroscience, 11, 843-850.

Susskind, J. M., Littlewort, G., Bartlett, M. S., Movellan, J., & Anderson, A. K. (2007). Human and computer recognition of facial expressions of emotion. Neuropsychologia, 45, 152-162.

Tipples, J. (2006). Fear and fearfulness potentiate automatic orienting to eye gaze. Cognition & Emotion, 20, 309-320.

Todd, R. M., Talmi, D., Schmitz, T. W., Susskind, J. M., Anderson, A. K. (2012). Psychophysical and neural evidence for emotion-enhanced perceptual vividness. Journal of Neuroscience, 32, 11201-11212.

Todorov, A., Mandisodza, A. N., Goren, A., & Hall, C. C. (2005). Inference of competence from faces predict election outcomes. Science, 308, 1623-1626.

66

Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale, R. J. Mansfield (Eds.) Analysis of Visual Behavior (pp. 549-586). MIT Press, Cambridge.

Vuilleumier, P., Armony, J. L., Driver, J., & Dolan, R. J. (2001). Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron, 30, 829-841.

Vuilleumier, P., Armony, J. L., Driver, J., & Dolan, R. J. (2003). Distinct spatial frequency sensitivities for processing faces and emotional expressions. Nature Neuroscience, 6, 624-631.

West, G. L., Anderson, A. K., Bedwell, J. S., & Pratt, J. (2010). Red diffuse light suppresses the accelerated perception of fear. Psychological Science, 21, 992-999.

Whalen, P. J. (1998). Fear, vigilance, and ambiguity: Initial neuroimaging studies of the human amygdala. Current Directions in Psychological Science, 7, 177-188.

Whalen, P. J., Kagan, J., Cook. R. G., Davis, F. C., Kim, H., Polis, S., . . . Johnstone, T. (2004). Human amygdala responsivity to masked feaful eye whites. Science, 306, 2061.

Whalen, P. J., Rauch, S. L., Etcoff, N. L., McInerney, S. C., Lee, M. B., & Jenike, M. A. (1998). Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. Journal of Neuroscience. 18, 411–418.

Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: I. Fitting, sampling, and goodness of fit. Perception and Psychophysics, 63, 1293-1313.

Willenbockel, V., Sadr, J., Fiset, D., Horne, G. O., Gosselin, F., & Tanaka, J. W. (2010) Controlling low-level image properties: the SHINE toolbox. Behavioral Research Methods, 42, 671-684.

Young, A. W., Rowland, D., Calder, A. J., Etcoff, N. L., Seth, A., & Perrett, D. I. (1997). Facial expression megamix: tests of dimensional and category accounts of emotion recognition. Cognition, 63, 271-313.

Zebrowitz, L. A. (1997). Reading Faces: Window to the Soul? Boulder, CO: Westview.

67

Appendix A: Stimulus sets

Figure A1: Schematic eyes used in Experiment 3. Schematic eyes were modeled from 19 exemplars (rows) posing disgust (leftmost column) and fear (rightmost column). The two intermediate sizes for each exemplar were created as linearly interpolated steps in vertical eye aperture from disgust to fear.

68

Figure A2: Chimeric expressions used in Experiment 7. 36 Chimeric expressions were created by combining the eyes (columns) and mouth regions (rows) of the 6 basic emotions.