The Nature of Mental Representations

How is knowledge represented in the Mind?

Think carefully about the people depicted.

What do they look like?

Now compare each of their voices

How do these representations exist in your mind?

The Nature of Mental Representation

What does your representation of a penny look like?

One of the greatest problems remaining is understanding the nature of mental representation

Knowledge Structures:Declarative Memory: facts that can be stated (Knowing that)

Procedural knowledge: procedures that can be implemented(Knowing how)

How to study mental representation…*Experiments manipulating represented knowledge*Neuropsychology

Cognitive MapsTolman 1930’s

Experiments by Tolman demonstrated that “representations” exist

• Group 1: Rats had to learn maze, reward for getting from start box to end box was food– Made more errors than group 1

• Group 2: no reinforcement for getting to end box– These rats learned the maze without making errors

• Group 3: no reward for 10 days. On 11th day food placed in end box for first time. – With one reinforcement ran maze as well as group 1

What do Tolman’s Maps look like?

Edward C. Tolman (1886-1959)

Latent learning goes against standard behavioristic principles, which claim that learning comes only from outcomes

Cognitive MapsTolman 1930’s

• Conclusions: Learning a cognitive map: – An internal representation of the maze

Cognitive MapsBees

vonFrisch 1967• Studied behavior of bees returning to hive after

locating nectar• Can form imaginal maps• Can use a symbolic form of communication• Different patterns of dances represent different

meanings

Cognitive MapsBees

vonFrisch 1967• Round dance: source less

than 100 yards from hive• Figure 8 dance: greater

distances

Propositional versus

Analogical Representation

The Age-old Debate

Representational Systems Require:

An orderly mapping between the Represented and Representing worlds

A mental representation is a system of symbols, conscious or unconscious, that are isomorphic to some aspect of the environment, used to make behavior-generating decisions that anticipate events and relations in that environment

C.R. GallistelRutgers University

“Words are the images of things”

The Greek poet Simonides, c. 500 B.C.

How about Book?

“Book” = “Libro”

Theories of Mental Representation

Meaning is grounded in the sensorimotor processes that result from interactions with the world. Semantic representations retain properties of the sensorimotor states that gave rise to them.

Analogical, Modal, & Embodied Meaning:

Meaning is represented by arbitrary codes that do not retain any of the properties of sensorimotor states that gave rise to them.

We don’t store images. Representations resemble abstract form of a preposition. Stored in terms of their deep meanings

Relationship between elements/ Element/ Object

Propositional & Amodal

Propositional

“Hammer” = ?“xuz1245”

Analogical

“Hammer” = ?

Propositional Representations for Circle:

“A series of points all of which are equidistant from a central point”

(Arbitrary code = English, Geometry)

“101101101101101010” (Arbitrary code = Binary Digit)

“x2 + y2 = r2” (Arbitrary code = Analytical Geometry)

Analogical Representation for Circle:

Analogical Representations:

Thermometers

Photographs

Cognitive mapsHumans

• Landmark knowledge: features: imaginal and propositional

• Route road knowledge: specific pathways: procedural and declarative knowledge

• Survey knowledge: estimated distances between landmarks: imaginally or propositionally(number)

Cognitive mapsHumans

Imagery

• Mental representation of things not currently being sensed by the sense organs– Often a recollection of something

• Can imagine things never experienced– Imagine what it would be like to jump from a plane

• Can imagine things that don’t exist– Imagine that Bush wasn’t the current president

• Can use any of the sensory modalities– Most research targets visual imagery

Kosslyn:We use imagery to solve problems

• Which is a darker red: An apple or a cherry?

• How many windows in your bedroom?

• How do you get to the subway from here?

EXPERIMENTS supporting ANALOGICALREPRESENTATIONS

Shephard and Metzler Experiments (1971) Rationale: If mental representations share features with their

referent, that is, they are analogical or isomorphic, thenrotating a mental image should take as much time as rotating

one physically.

Hypothesis: When people rotate a mental image, the farther they have to rotate it, the longer they will take.

Method: Subjects were shown pairs of two-dimensional drawings of three-dimensional objects and were asked whether the drawings in each pair represented the same

object in a different orientation or whether they represented different objects.

Results: 97% Accuracy. The time it took the subjects to provide the answer was proportional to the angular

difference between the two drawings.For example, if the second image was rotated 60 degrees,

the subject would take twice as long than if it was just rotated 30 degrees.

Experiment

• Close eyes and imagine a cube floating in space in front of you

• Now: Grasp the Left Front bottom corner of the cube with your left hand and grasp the right back top corner of the cube with your right hand.

• Now Rotate the cube so that the corner in your left hand is directly below the corner in your right hand

• Describe

Using positron emission tomography (PET), an axial brain slice taken 56 mm above the anterior commissure–posterior commissure (AC–PC) line shows activation in the primary motor cortex (M1) when data from the external-action condition were subtracted from those of the internal-action condition. Depending on the strategy used, motor regions of the brain are recruited during mental rotation. The result also shows that the strategy used to accomplish a given task can vary according to previous training, and can be adopted voluntarily.

Other evidence suggesting that representations are analogical:

1) Mental images of large objects (e.g., an elephant) take longer to scan from left to right than smaller objects

2) Details are more difficult to work out in mental images of small things (fruit fly) than in large objects (airplane). For example, Elephant head to tail last longer than Pig head to Pig tail.Similar with map experiments.

Distance in image is proportional to time spent scanning

Kosslyn 1975

• Imagine :• elephant and a rabbit• Rabbit and a fly• Rabbit and elephant sized fly • Rabbit and a fly sized elephant• TASK: describe details of objects• RESULTS: took longer to describe smaller object

of pair– =Perceptual resolution better for larger objects

Evidence for Analogical Scanning

Stephen Kosslyn (right) and one of his collaborators

• Perception: longer to scan longer distance than short distances

• Experiment to test imaginal scanning– Shown a map of an imaginary island with

landmarks– Studied until could reproduce from memory– Critical phase

• Hear name of an object (tree)• Picture map, mentally scan directly to object and

press a key when have arrived to the object• Hear name of second object, scan to that object’s

location• Measured RT

Image scaling

Findings Distance & RT are Positively Related

Image scaling

• Spatial representations can be scanned in the same way that physical percepts can be scanned

• Our strategies and responses for imaginal scanning are expected to be functionally equivalent to those we use for perceptual scanning

Auditory imaginal representation

• Intons-Peterson et al 1992– Subjects take longer to mentally shift a sound

upward in pitch from the low purring of a cat to a high pitched ringing of a phone than to switch from purr to clock tick.

Zenon Pylyshyn

Problem: Is it all due to task instructions?Demand Characteristics

What is farther west, Reno or San Diego?

Is the mental image of Kosslyn’s island really map-like?

Neuropsychological evidence

• Georgopoulos et al 1989– Single cell recordings in motor cortex of monkeys

– Monkey trained to physically move a handle perpendicular to and counterclockwise to a target light

– During rotation cortical activity recorded

– Later take away handle, cortical activity recorded

– RESULTS: activity in motor cortex showed same cortical cells respond when monkeys are anticipating the movements of the rotations

L.H.Farah’s patient

• Visual imagery(color and shape) distinct from spatial imagery(depth and distance)

• LH head injury at age 18• Lesions in left temporo-occipital regions, right

temporal, right inferior temporal• Anomia- could see but could not apply verbal

label– Bad at describing color and shape imaginal info– Good with spatial manipulations

What processes do we use to represent and manipulate

knowledge?• How we represent info influences how

easily we can perform cognitive tasks

• CMLIX*LVIII

What processes do we use to represent and manipulate

knowledge?• How we represent info influences how

easily we can perform cognitive tasks

• CMLIX*LVIII

• Easier as 959*58

Declarative vs. Procedural

• Declarative: knowing that

• Procedural: knowing how

• Write normal signature as quickly as possible: From first letter of first name to last letter of last name.

• Now turn over paper– Write your signature backward starting with

last letter of last name to first letter of first name

Declarative vs. Procedural

• For both signatures had declarative knowledge about what comes next to each letter

• Only had procedural for the normal writing of name

Concepts

• Idea about something = fundamental unit of symbolic knowledge

• Relates to other concepts– Red, round, fruit

• How do we organize concepts– Categories: organization based on common

features or similarity to a prototype

Schemas

• Can include other schemas

• Encompass typical general facts that can vary from one instance to another

• Vary in their degree of abstraction

• Often have if, then relationships embedded– If a glass falls on to a hard surface, then…

• It should break

Schematic relationships

• Young child calls out “mama!” after he falls from the jungle gym at the playground

• Who is he calling to- a 75 yr old woman, a 45 year old man, and a 30 year old nun or 25 year old woman?– Utilize schemas to answer this questions

Procedural knowledge

• Problem solving• Improved by practice until becomes automatic• Others have made a distinction from declarative

knowledge as “non declarative”– Includes perceptual motor and cognitive skills:

procedural

– Simple associative knowledge: CC

– Priming

Experiment1/2 class close eyes

• ZAZIP

• GASPETHIT

• POCH YUSE

• OWCH MINE

• ILCHI

• ACOT

Other half of class close eyes

• TECKAJ

• STEV

• ASTEREW

• OLACK

• ZELBAR

• ACOT

• PIZZA• SPAGHETTI• CHOP SUEY• CHOW MEIN• CHILI

• TACO OR COAT

• JACKET• VEST• SWEATER• CLOAK• BLAZER

• TACO OR COAT

Declarative/ Procedural differences

• Amnesiacs: can learn new skill but can’t express, can’t consciously remember that they have been learning it.

• Show improvements in

skill tasks: mirror drawing

• One representation is part of non declarative knowledge, trying to make it explicit can be very difficult

Karl S. Lashley and the Problem of Serial Order

Musicians play at a rate so fast that feedback cannot serve as a stimulus for the next note. Pigeons were

taught to learn an arbitrary sequence of colors:

redgreenyellowblue

Successive Chaining paradigm:S(A):R(A) S(B):R(B)S(C):R(C)S(D):R(D)

In successive chaining, each cue is responded to individually

Serial Order & Its Representation

No Possibility of Chaining

Simultaneous Chaining Paradigm:S(A):S(B):S(C):S(D) R(A) R(B)R(C)R(D)

All items are presented simultaneously, and there is no differential feedback during the execution.

Herb Terrace and one of his students at Columbia University

Knowledge of the Ordinal Position of List Items inRhesus Monkeys

Chen, Swartz, & TerracePsychological Science, 1997, pp. 80-86.

LIST1. A1B1C1D12. A2B2C2D23. A3B3C3D4

RESPONSEBirdFlowerFrogShellsTreeWeaselFlyWater

ElkRocksLeavesPerson

NEW LIST1. A2B4C1D32. B3A1D4C23. D1C3B2A4

NEW RESPONSETreeFishFrogPerson

RocksBirdTomatoFlyShellsLeavesWeaselSky

Results: It takes longer for Rhesus monkeys to learn a new list if items are repeated in a different ordinal position.This provides evidence that Rhesus monkeys internally represent ordinal position

Still, one of the greatest problems confronting psychology is the nature of mental representation

http://www.psych.uni.edu/psychexperiments/Exps/labexperiments.htmhttp://www.cog.brown.edu/~tarr/stimuli.html www.sims.berkeley.edu/~sack/ CM/detailed-intro.htm