Holistic, part-based and semantic processes in face recognition:

Dissociations and interactions

¹ Department of Psychology, University of Toronto

² Rotman Research Institute, Baycrest Centre for Geriatric Care

³ Centre for Vision Research, York University

In collaboration with:

David AnakiJosee Rivest

Sandra BlackLarry LeachYakir KaufmanBruce Bolster

• Historical background

• Enduring problems

• New studies

• Dissociations among three agnosic patients

• Prosopagnosia (patient DC)

• Object agnosia (patient CK)

• Pure alexia (patient CM)

• Semantic/associative prosopagnosia (patient AP)

• Interactions among face, object, and semantic systems

Outline

Historical background

Bodamer, 1947

Definition: prosopagnosia

Acquired inability to recognize familiar faces from vision

• Recognition triggered by alternative means: ex.: voice, paraphernalia, gait

• Appropriate semantic information about familiar people

• Good visual acuity • Face recognition may be achieved by a feature-by-

feature approach (?)

Must NOT be secondary to:• sensory loss • general intellectual loss • memory & language problems • or a combination of these factors

RECOGNITION impairment...

Associated Recognition Deficits

• Object agnosia

• Pure alexia • Good visual acuity • Good semantic memory/processing• Appropriate orthographic units • No agraphia, no aphasia • Letter-by-letter readers

But again… dissociations:

• Prosopagnosia with alexia but without object agnosia

(e.g. Buxbaum, Glosser & Coslett, 1998, De Renzi & di Pellegrino, 1998)

• Prosopagnosia with object agnosia but without alexia

(e.g. Humphreys & Riddoch, 1989)

• Object agnosia without prosopagnosia or alexia

(e.g. Humphreys & Rumiati, 1998; Rumiati, Humphreys, Riddoch & Bateman, 1994)

+

Neuroanatomy

From patients…

• Bilateral damage in the occipitotemporal cortices:- the inferior temporo-occipital regions - the lingual and fusiform gyri (e.g. Damasio, Damasio & Van Hoesen, 1982;

Takahashi, Kawamura, Hirayama, Shiota & Isono, 1995)

• A right hemisphere lesion (in the equivalent areas) - may be sufficient (e.g. De Renzi, 1986a; Landis, Regard & Bliestle, 1988; Rapcsak, Polster,

Comer & Rubens, 1994; Takahashi et al., 1995)

More evidence…

“Normal” human brains: fMRI studies

- the fusiform gyri (Face Fusiform Area: FFA) (e.g. Kanwisher, Woods, Iacoboni & Mazziotta, 1997)

A PET study- the medial fusiform gyri (areas 19-37) - the right parahippocampal gyrus (area 36) - the anterior region of the temporal poles (Sergent & Signoret, 1992)

1. Modularity vs. General Purpose2. Representations vs. Processes3. Face-specific vs.Individuation or Expertise4. Focal/Localized vs. Distributed5. Laterilzed vs. Bilateral6. Holistic vs. Analytic or part-based7. Different versions of holistic

Enduring Debates: Are faces special?

CK - Object Agnosia & Alexia

Behrmann, Moscovitch & Winocur (1994)

Behrmann, Winocur & Moscovitch (1992)

Moscovitch, Winocur & Behrmann (1997)

• 41-year old man• 16 years education (MA degree)• Manager • Closed-head injury: hit by a car when jogging• Bilateral thinning in the occipito-temporal regions

Behrmann, Moscovitch & Winocur (1994)

Behrmann, Winocur & Moscovitch (1992)

Moscovitch, Winocur & Behrmann (1997)

• Word recognition (reading)

• Object recognition• Residual blindness

in upper left field

IMPAIRED

• Average IQ • Acuity• Upright face recognition• Visual construction abilities• Memory • Semantic knowledge

NORMAL

CK - Object Agnosia & Alexia

Face module

• Features - primarily internal

• At least two of three needed

• Orientation specific - 0 (upright) to 60-90 degrees

• Global configurational properties– template and/or second order relations

Face counterpart in object system

• Feature - internal and external

• Distinctive? Verbalizable?

• Orientation sensitive

• Categorical or local relations (jig-saw effect)

• Integration of features and relations

Interaction of face and face counterpart

• Mapping of features and relations in counterpart to holistic face representations stored in long-term memory (FRUs)

• Interactive activation between systems: face, object, and semantics

Part-based

OBJECTS WORDS

Holistic

FACES

Upright faces Inverted faces

Double dissociation

Words X X

CK DC

Faces

Objects

√

X √

X

√

Recognition CM

√

√

Patients

More dissociations

CK DC

Faces

Objects

Words

√

X

X

√

X

√

Recognition CM

√

X

√

Patients

CK DC

Faces

Objects

Words

√

X

X

√

X

√

Recognition CM

√

X

√

Patients

Prosopagnosia

Object Agnosia& Alexia

Pure Alexia

DC - Prosopagnosia

• 54-year old man• 16 years education• Detective for the homicide division of a police

force (on disability)• Colloid cyst hydrocephalus• Posterior cerebral artery infarction

DC - Prosopagnosia

• Bilateral lesions in the lingual

gyri, Brodmann Areas 18 & 19

R L

DC - Prosopagnosia

• Bilateral lesions in the lingual gyri: more atrophy to the right fusiform gyrus than the left

R L

• Superior IQ • • AcuityAcuity• Object recognition• Reading • Visual construction abilities• Space perception • Colour vision• Memory • Semantic knowledge

• Face recognition• Bilateral superior

field deficits

IMPAIRED NORMAL

DC - Prosopagnosia

Face recognition:

Famous faces

slow —no instant recognition Looks like a “piecemeal” approach

Bob Hope: “I will guess that it is Bob Hope. He has a long nose, his hair cut is him too, a side ways grin; I am not 100% sure that it is him...”

Hollywood stars• Jerry Seinfeld: “I do not know, an ordinary guy” “Oh, Jerry”

—watched Seinfeld at least once a day • Denzel Washington: “a US ball player”

—described many movies that he was acting in • Michelle Pfeiffer: “A model: Sharon Stone? Christie Brinkley?”

His family members and myselfwith hair covered:

• misidentified his son (who was in the testing room)• misidentified me

CM - Pure Alexia

• 35-year old man• 15 years of education• Computer programming• Closed-head injury / drugs• Temporal occipital abnormalities on EEG

CM - Pure Alexia

IMPAIRED NORMAL

• Word recognition (reading): Letter-by-letter reader

• High average IQ• Acuity• Face recognition• Object recognition• Visuo-spatial abilities

Is there one part-based system?

OBJECTS WORDS

Holistic

FACES

Processing systems

Part-based

Are there separate part-based systems?

OBJECTS WORDS

Holistic

FACES

Processing systems

Part-based Part-based

Does part-based face recognition depend on one or both part-based systems?

OBJECTS WORDS

Holistic

FACES

Processing systems

Part-based Part-based

CK - Object Agnosia & Alexia

Moscovitch, Winocur & Berhmann (1997)

Part-based

OBJECTS WORDS

Intact upright face recognition

Holistic

FACES

Part-based

CM - Pure Alexia

OBJECTS WORDSFACES

Part-based Part-basedHolistic

Intact face and object recognition

Can part-based face recognition rely only on systems mediating recognition of objects and words?

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

Or does it rely on the interaction between part-based and holistic systems ?

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

DC - Prosopagnosia

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

Intact object and word recognition

CK CM DC

Faces

Objects

Words

√

X

X

√

X

√ √

X

√

Recognition

Participants

Brain-damaged Control

DCB

√

√

√

Controls

n=12

√

√

√

• Upright faces

Tasks: Face recognition

Mean Number of Famous People Recognized from Photos (Max. = 70)

0

10

20

30

40

50

60

70

Controls CK DC DCBCM

perfect score

Mea

n N

umbe

r of

Peo

ple

Rec

ogni

zed

Results

Caricatures of Famous People

Ronald ReganWoody Allen

Caricatures of Famous People Recognized (Max. = 29)

Perc

ent m

ean

reco

gnit

ion

0102030405060708090

100

Controls CK DC DCB

Perc

ent c

orre

ct id

enti

fica

tion

0102030405060708090

100

Controls CK DC DCB

Mothers

Fathers

Child

Percent of Correct Responses on a Seven-Item Forced Choice Recognition Test for Identifying Each of the

“Parent” Photos from the Photo of a “Child”

0102030405060708090

100

DC DCBControls CK CM

Perc

ent m

ean

reco

gnit

ion

Tasks: Face recognition

• Fractured • Disguised

Faces

• Inverted

Who is:

• InvertedJason Alexander

George

Who is:

• Fractured Oprah

Who is:

• Disguised Eddie Murphy

Perc

ent R

ecog

nize

d

Results

Controls CK DC DCBCM0

102030405060708090

100

Percentage of Correct Recognition of Inverted Faces

Percentage of Correct Recognition of Fractured Faces

Results

0102030405060708090

100Pe

rcen

t R

ecog

nize

d

Controls CK DC DCBCM

0102030405060708090

100Pe

rcen

t Rec

ogni

zed

Percentage of Correct Recognition of Disguised Faces

Controls CK DC DCBCM

Results

Summary of findingsDamaged systemsFace

recognitiontasks

Object & Word

Disguised

Upright

Fractured

Inverted

√

X

√

X

Face

X

X

X

X

Word

√

√

√

√

There are separate part-based systems

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

Only the part-based object system contributes to some aspects

of face recognition

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

It does so by interacting with an intact upright face-system

Percent of Upright Cartoon Faces Identified Correctly (Max. = 31)

0102030405060708090

100

Controls CK CM DC DCB

Perc

ent m

ean

reco

gnit

ion

Percent of Inverted Cartoon Faces Identified Correctly (Max. = 31)

0102030405060708090

100

DC DCBControls CK CM

Perc

ent m

ean

reco

gnit

ion

Internal Features Inverted

Jacqueline Kennedy Robin Williams

Mean Correct Recognition of Photos of Famous People When Upright and Intact and

When Internal Features Were Inverted

Internal Inverted (Max. = 11) Intact (Max. = 11)

0

2

4

6

8

10

DC DCBControls CK CM

0

2

4

6

8

10

DCBControls CK CM DC

Mea

n C

orre

ct R

ecog

niti

on

External Features Inverted

Meg Ryan

Mean Correct Recognition of Photos of Famous People When Upright and Intact and

When External Features Were Inverted

External Inverted (Max. = 12) Intact (Max. = 12)

0

2

4

6

8

10

12

DC DCBControls CK CM

0

2

4

6

8

10

12

DC DCBControls CK CM

Mea

n C

orre

ct R

ecog

niti

on

Semantic system

OBJECTS WORDS

holistic

FACES

Recognition

Part-based

Structural representation

Recognition

Semantic system

OBJECTS WORDS

holistic

FACES

Part-based

Corresponding structural representation

Misaligned Along the Vertical Midline

Elizabeth Taylor

Mean Correct Recognition of Photos of Famous People That Are Intact or Misaligned Along the Vertical Midline

(Max. = 10)

Misaligned Vertical Intact

10

0

2

4

6

8

10

DC DCBControls CK CM

0

2

4

6

8

DC DCBControls CK CM

Mea

n C

orre

ct

Misaligned Along the Horizontal Midline

Madonna

Mean Correct Recognition of Photos of Famous People That Are Intact or Misaligned Along the Horizontal Midline

(Max. = 10)

Misaligned Horizontal Intact

10

0

2

4

6

8

DC DCBControls CK CM

0

2

4

6

8

10

DC DCBControls CK CM

Mea

n C

orre

ct

Dan Akroyd

Mean Correct Forced-Choice Recognition of Single Isolated Face Parts (Eyes, Nose, or Mouth)

for Identified Faces of Famous People (Max. = 15)

Identified Recognized Part

0

2

4

6

8

10

12

14

DC DCBControls CK CM

0

2

4

6

8

10

12

14

DC DCBControls CK CM

Mea

n C

orre

ct R

ecog

niti

on

Mean Correct Forced-Choice Recognition of Single Isolated Face Parts (Eyes, Nose, or Mouth)

for Non-Identified Faces of Famous People (Max. = 15)

Non Identified Recognized Part

0

2

4

6

8

10

12

14

DC DCBControls CK CM

0

2

4

6

8

10

12

14

DC DCBControls CK CM

Mea

n C

orre

ct R

ecog

niti

on

Bev Doolittle: The forest has eyes

CK - Object Agnosia & Alexia

OBJECTS WORDS

Intact upright & disguised face recognition

Holistic

FACES

Part-based Part-based

DC - Prosopagnosia

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

Impairment of all aspects of face recognition

Damage to the holistic face system leads to global face recognition deficits

OBJECTS WORDS

Holistic

FACES

Part-based Part-based

Face Representation

Depth segregation and “closure” for faces and objects

Mooney Faces and Objects

Correct Identification of Mooney Faces and Objects

Faces (Max. = 7) Objects (Max. = 9)

0

2

4

6

DC DCBControls CK

0

2

4

6

8

DC DCBControls CK

Mea

n C

orre

ct I

dent

ific

atio

n

Depth Segregation for Faces and Objects

William Shakespeare

Pierre E. Trudeau

Michael Jackson

Bob HopeEddie Murphy

Percent Correct Identification and Forced Choice Recognition of Overlapping Caricatures of Famous People (Max. = 27)

Identification Forced Choice Recognition

0

20

40

60

80

100

Controls CK DC DCBCM

0

20

40

60

80

100

Controls CK DC DCBCM

Perc

ent C

orre

ct

ArcimboldoIdentity of Composite

Faces (i.e. made of objects)