Shepard (1984) Ecological constraints on internal ...wexler.free.fr/library/files/shepard (1984)...

Psychological ReviewVOLUME 91 NUMBER 4 OCTOBER 1984

Ecological Constraints on Internal Representation: ResonantKinematics of Perceiving, Imagining, Thinking, and Dreaming

Roger N. ShepardStanford University

This article attempts a rapprochement between James Gibson's ecological opticsand a conviction that perceiving, imagining, thinking, and dreaming are similarlyguided by internalizations of long-enduring constraints in the external world.Phenomena of apparent motion illustrate how alternating presentations of twoviews of an object in three-dimensional space induce the experience of thesimplest rigid twisting motion prescribed by kinematic geometry—provided thattimes and distances fall within certain lawfully related limits on perceptualintegration. Resonance is advanced as a metaphor for how internalized constraintssuch as those of kinematic geometry operate in perception, imagery, apparentmotion, dreaming, hallucination, and creative thinking, and how such constraintscan continue to operate despite structural damage to the brain.

Oxford philosopher of science Rom Harrein his book Great Scientific Experiments:Twenty Experiments That Changed our Viewof the World (Harre, 1983) includes James J.Gibson's work on perception along with ex-

This article, which I dedicate to the memory of JamesJ. Gibson, is an expanded version of the Gibson MemorialLecture, which I gave at Cornell University on October21, 1983. I thank the members of the Department ofPsychology at Cornell for providing me with an oppor-tunity to clarify the relation of my thinking to Gibson's,and the National Science Foundation for supporting boththe preparation of this article and most of the researchon which it is based (especially through Grants GB-31971X, BNS 75-02806, and BNS 80-05517).

The faults that undoubtedly remain have at least beengreatly reduced as a result of the helpful suggestionsmade by numerous colleagues including Fred Attneave,Maya Bar-Hillel, Lynn Cooper, Joyce Farrell, John Flavell,David Foster, Jennifer Freyd, Randy Gallistel, FrankKeil, Edward Kessler, Carol Krumhansl, Laurence Ma-loney, Ann O'Leary, Edward Oshins, Herbert Simon,Elizabeth Spelke, Richard Thompson, Brian Wandell,Benjamin White, and, especially, Gerald Balzano, JamesCutting, Julian Hochberg, Michael Kubovy, and UlricNeisser. Each of these last five contributed extraordinarilypainstaking, thoughtful, and enlightening comments.

Requests for reprints should be sent to Roger N.Shepard, at Department of Psychology, Uris Hall, CornellUniversity, Ithaca, New York 14853, 1984-1985.

periments by such giants of the natural sci-ences as Aristotle, Galileo, Newton, Boyle,Lavoisier, Rutherford, and Pasteur. Countingmyself among students of perception whohave come to recognize the challenge thatGibson posed to many long-accepted ideas, Ihave been moved to work out how the essen-tial insight that informs Gibson's ecologicalapproach might be extended into a realmthat has been for me of great and continuinginterest.

My efforts in this direction have not pro-ceeded without trepidation. Gibson himselfis widely considered to have regarded thisrealm as insignificant or, worse, nonexistent.I refer to the realm of what I have calledinternal representation (Shepard, 1975; She-pard & Chipman, 1970). Even in my title,which begins auspiciously enough with thatgood word ecological, I have risked anathemaby moving immediately on to those verywords internal representation. How can onewho finds Gibson's insight into perception tobe so congenial persist in exploring the ap-plication of this insight to a realm that Gibsonhimself never countenanced? At least part ofthe answer must be that even investigatorswho agree that they are studying perception

Copyright 1984 by the American Psychological Association, Inc.

417

418 ROGER N. SHEPARD

may be found, on closer examination, to havequite different objectives.

Differing Goals in the Study of Perception

Goal of Understanding a Sensory Organ'sTransduction of Incident Energy

Those who call themselves psychophysicistsor vision researchers tend to seek laws relatingjudgments about sensory events to physicallymeasurable properties of proximal stimuli,and those who call themselves sensory psy-chophysiologists seek, in addition, relationsof these kinds of variables to physically mea-surable activities within the nervous system.The primary goal for both of these classes ofresearchers seems to be the elucidation of themechanisms whereby energy impinging on asensory organ is transduced into neural activ-ity and thence into behavior.

Goal of Understanding an Organism'sPerception of its Environment

Helmholtz (1856/1962, chap. 1), whilepursuing the goal of understanding sensorytransduction of proximal stimulation, alsorecognized that an organism must interactappropriately with distal objects in its envi-ronment. Yet this latter, ecologically orientedobjective was not fully articulated as theprimary goal of the study of perception untilBrunswik (1956) and J. Gibson (1950)stressed that as the organism, objects, andsources of illumination move about in space,the variations in proximal stimulation bearlittle resemblance to the particular unidimen-sional variations of retinal size, brightness,wavelength, or duration that psychophysicistsand psychophysiologists have typically ma-nipulated in their laboratories.1

True, early investigators such as Hering(1878/1964), Mach (1886/1959), and evenHelmholtz (1856/1962) suggested that theflux of proximal stimulation does containsome features that are invariantly related todistal objects. For example, although the lightenergies reaching the eye from two surfacesof different reflectances vary widely withchanges in illumination, the ratio of thosetwo energies remains constant. Then Cassirer(1944) explicitly introduced the mathematicalconcept of invariance over a group of trans-

formations as a characterization of such per-ceptual constancies. Nevertheless, it remainedfor Gibson to adopt the radical hypothesis ofwhat he called the ecological approach toperception (Gibson, 1961, 1979), namely, thehypothesis that under normal conditions, in-variants sufficient to specify all significantobjects and events in the organism's environ-ment, including the dispositions and motionsof those objects and of the organism itselfrelative to the continuous ground, can bedirectly picked up or extracted from the fluxof information available in its sensory arrays.

In the case of the modality that mostattracted Gibson's attention—vision—the in-variants generally are not simple, first-orderpsychophysical variables such as direction,brightness, spatial frequency, wavelength, orduration. Rather, the invariants are what J.Gibson (1966) called the higher order featuresof the ambient optic array. (See J. Gibson,1950, 1966, 1979; Hay, 1966; Lee, 1974;Sedgwick, 1980.) Examples include (a) theinvariant of radial expansion of a portion ofthe visual field, looming, which specifies theapproach of an object from a particulardirection, and (b) the projective cross ratiosof lower order variables mentioned by J.Gibson (1950, p. 153) and by Johansson, vonHofsten, and Jansson (1980, p. 31) and in-vestigated particularly by Cutting (1982),which specify the structure of a spatial layoutregardless of the observer's station point.

For invariants that are significant for aparticular organism or species, Gibson coinedthe term affordances (J. Gibson, 1977). Thus,the ground's invariant of level solidity affordswalking on for humans, whereas its invariantof friability affords burrowing into for molesand worms. And the same object (e.g., a woolslipper) may primarily afford warmth of footfor a person, gum stimulation for a teethingpuppy, and nourishment for a larval moth.The invariants of shape so crucial for theperson are there in all three cases but are lesscritical for the dog and wholly irrelevant forthe moth.

1 Correspondingly, I have elsewhere argued for a kindof psychophysics that does not restrict itself to theconsideration of proximal variables (see Shepard, 198 la,1981b, 1982a).

ECOLOGICAL CONSTRAINTS ON INTERNAL REPRESENTATION 419

Although the goal of identifying the in-variants in the optic array that correspond toall such affordances is far from having beenattained (Hochberg, 1982; Neisser, 1977),progress has been made in identifying theinvariants underlying the perception of indi-vidual human gaits (Cutting, Proffitt, & Koz-lowski, 1978; Kozlowski & Cutting, 1977)and of age in human and animal faces (Pit-tenger & Shaw, 1975; Shaw & Pittinger, 1977),and in establishing that the ability to pick upsuch invariants as rigidity versus nonrigidityemerges early in human infancy (E. Gibson,1982; E. Gibson, Owsley, & Johnston, 1978;E. Gibson & Spelke, 1983; Spelke, 1982).

According to James Gibson, the notion—widely accepted since Helmholtz—that wemust construct our percepts by combiningsensory cues was a misguided consequenceof elementaristic, ecologically invalid labo-ratory experiments in which, for example, aphysically restrained observer was permittedonly a brief, monocular glimpse of the stim-ulus. In natural settings we enjoy binocularity,free mobility, and persisting illumination. Inthat case, Gibson claimed, no inference isrequired because invariants in the shiftingoptic array uniquely specify the layout of theenvironment.

Goal of Understanding the Capabilities ofan Organism Under Reduced Circumstances(of Incomplete Information, InsufficientTime, or Damaged Brain)

Even those who follow Gibson this far inpursuing the goal of understanding how or-ganisms function in their natural environmentmay nevertheless disagree about what to in-clude under its heading. For most of thosewho follow the ecological approach, the goalhas been confined to the identification andspecification of the invariants that are suffi-cient for the veridical perception of the localenvironment under favorable conditions ofvisibility, mobility, and neural integrity. Theyhave manifested little interest in three otherkinds of questions: First (noted, e.g., by Ull-man, 1980), they have not pursued questions,raised by students of neurophysiology andartificial intelligence, concerning the mecha-nisms that enable an individual to extract theappropriate invariants from the informationavailable at its sensory surfaces.

Second, they have neglected questions,raised by students of cognitive science, con-cerning how we know about (a) objects, re-lations, and events that are obscured by dark-ness or by obstructed, monocular, brief, orintermittent access and also (b) those thatare beyond the region that is directly affectingus during a given period of time. There oftenis no information in sensory arrays aboutevents that have occurred in the past, thatare occurring in another place, that willoccur in the future, or that might occurunder altered circumstances, even thoughsuch events can be of great importance andcan be known to us in our natural environ-ment.

Third, students of ecological optics haveignored questions, raised by experimentalcognitive psychologists and by clinical neu-rologists, concerning what happens when theinformation available in the sensory arrays—although sufficient to specify the immediateenvironment—exceeds the processing capa-bilities of the individual. I argue that thereare limits on the intervals of space and timeover which we can integrate informationavailable in the sensory arrays and that theselimits are themselves lawful in ways that cryout for explanation. Moreover, there arequestions of what happens when this pro-cessing capability is further reduced as aresult of brain damage, which also occurs inour natural environment as a result of injury,disease, or (as I am increasingly reminded)advancing age. Why do brain lesions lead toparticular perceptual dysfunctions, and espe-cially, how can the brain often reestablishmore or less normal functioning despite suchlesions?

In short, although I agree with Gibson thatthe brain has evolved to extract invariantsunder favorable conditions, I also presumethat it has evolved to serve the organismunder less favorable conditions of nighttime,obstructed, and spatially or temporally limitedviewing and, even, of structural damage tothe brain itself.

Proposed Extension of theEcological Approach

In striving to accommodate questions fromall three classes just mentioned, withoutabandoning Gibson's essential insight, one


seemingly has to come to terms with therelation between the organism's representationof objects that are and those that are notimmediately affecting its sensory arrays, thatis, with the relation between perception andmental imagery.

Problem of Mental Imagery

I conjecture that Gibson disavowed theterm mental image because he could notimagine what sort of thing a mental imagecould be. He readily spoke of perceiving anobject, because that object is a physical thing.But in his view "the notion of 'mental images'as distinguished from 'material images' seemsto be wholly wrong" (J. Gibson, 1974, p. 42).On the one hand, if a mental image is not aphysical thing, what on earth is it?

We certainly do not summon up pictures inside our headfor they would have to be looked at by a little man inthe head. . . .Moreover, the little man would have eyesin his head to see with and then a still littler man andso ad infinitum, (J. Gibson, 1974, p. 42)

On the other hand, if a mental image is aphysical (i.e., neural) process in the brain,we must admit that we know next to nothingabout the process. Surely, what determineswhether an animal survives is its interactionwith its external environment, regardless ofwhich of the possible internal mechanismsfor mediating that interaction is realized inthat particular animal. However, in neglectingthe representation of objects and events thatare not physically present, Gibson seems tohave given up too much.

I proposed to accommodate mental imag-ery by saying that (a) imagining, like perceiv-ing, is surely performed by physical processesin the brain but (b) we do not need to knowany details of these processes in order tostudy imagining (any more than Gibson hadto have such knowledge in order to studyperceiving). What we imagine, as much aswhat we perceive, are external objects; al-though in imagining, these objects may beabsent or even nonexistent. We can thereforecarry out experiments on both perceptionand imagery by probing individuals withappropriately chosen external stimuli (Pod-gorny & Shepard, 1978, 1983; Shepard, 1975,1981b; Shepard & Chipman, 1970; Shepard

Figure 1. Displays used for the perceptual condition (Parta), the imaginal condition (Part b), and the ensuing testprobe (Part c) in one of the experiments by Podgornyand Shepard (1978). (From "The Mental Image" byR. N. Shepard, 1978, The American Psychologist, 33, p.133. Copyright 1982 by the American PsychologicalAssociation. Adapted by permission.)

& Cooper, 1982; Shepard & Podgorny, 1978;and, for a brief overview, Shepard, 1978c).

An experiment that Podgorny and I carriedout illustrates the point. On each trial, aperson looked at a square grid. In the percep-tual condition, some squares had been shadedto form a certain object (such as the blockletter F in Figure 1, Part a); in the imaginalcondition, no squares were shaded but theperson was asked to imagine that the samesquares had been shaded (Figure 1, Part b).In both conditions, we then flashed a coloredprobe dot in one of the squares (Figure 1,Part c) and measured the latency of theperson's response indicating whether the dotdid or did not fall on the (perceived orimagined) object. With experiments of thistype, we obtained two major results: First,the reaction times depended on the positionof the probe relative to the figural object ina way that implicates orderly constraints inthe perceptual mechanism. For example, re-sponses were consistently slower to probesthat were closer to boundaries between figuraland nonfigural squares. Second, the reactiontimes exhibited virtually the same pattern inthe imagery and the perceptual conditions,suggesting that the object was internally rep-resented in the same way regardless of whetherit was physically present or only imagined.(Podgorny & Shepard, 1978, 1983.)

Although I thus speak of internal represen-tations, I agree with J. Gibson (1970, p. 426)as well as with Neisser (1976, p. 57) that oneinvites unnecessary perplexities by speaking—as imagery researchers sometimes carelesslydo—of "seeing," "looking at," "inspecting,"or "rotating" one's images or internal repre-


sentations. Rather than say that one sees orrotates the image of an object (as if the imagewere itself a physical thing), one can avoidsuch perplexities by simply saying that oneimagines the object and/or its rotation (whichare potentially physical things). The distinc-tion is, for example—as Michael Kubovy(1983) has well put it—between the acceptableformula Imagine [Rotation of (Object)] andthe problematic Rotate [Image of (Object)}.On occasion, I have spoken of "experiencing"an image or similarly a percept, but only asa kind of shorthand for "undergoing thecorresponding (but largely unknown) physicalprocesses in the brain" (cf. Place, 1956;Smart, 1959). Properly speaking, our experi-ence is of the external thing represented bythose brain processes, not of the brain pro-cesses themselves. At the same time, by ac-knowledging that perceiving and imagining—as well as remembering, planning, thinking,dreaming, and hallucinating—do correspondto brain processes, we at least open the doorto possible connections with evolutionarybiology, clinical neurology, and artificial in-telligence.

Evolutionary Perspective on Perceptionand Representation

Whatever we possess in the way of a per-ceptual and/or representational system mustbe the product of a long evolutionary history.Our remote ancestors, like many survivingprimitive species (ranging from single-celledanimals to worms), could not extract higherorder invariants corresponding to distal ob-jects of the sort that usually concern us now.Instead, they proceeded on the basis of prox-imal stimuli of a chemical or mechanicalnature. Only with the evolution of increas-ingly powerful mechanisms for the processingof optical, acoustical, and tactual informationhave we gained access to remote objects andevents.

In keeping with the ecological approach, Ibelieve that (initially) the primary functionserved by this more sophisticated perceptualprocessing was to partition the informationavailable in these various incoming formsinto (a) the invariants uniquely correspondingto distal objects, events, and layouts, and (b)the complementary variables corresponding

to the moment-to-moment changes in thedisposition of those objects, events, and lay-outs, and of the self in relation to them. Sucha partitioning is now pervasive: We visuallyperceive both a persisting object and its cur-rent spatial relation to us. We also recognizeboth the face of a friend and its momentaryexpression, both what has been written andthe format in which it is written, both whathas been said and the emotional state of thespeaker, and both a particular melody andthe pitch height and timbre at which it hasbeen played.

However, this is not the end of the evolu-tionary story. As Gibson emphasized, higherorganisms are not merely observers; they areactive explorers and manipulators of theirenvironment. If such exploration and manip-ulation is not just random trial and error, itmust be guided by some internal schema(Hochberg, 1981, 1982; Neisser, 1976) orhypothesis (Krechevsky, 1932). At this point,a new type of function emerges that is relatedto perceptual and to motoric functions, butis not identical to either. I refer once againto the ability to remember, to anticipate, andto plan objects and events in their absence.The alternative claim (cf. Gibson, 1970), thatsuch functions are entirely separate fromperception, is untenable in view of experi-mental results of the sort reported by Pod-gorny and Shepard (1978, 1983) and others(as reviewed in Finke, 1980; Finke & Shepard,in press; Shepard & Cooper, 1982; Shepard& Podgorny, 1978). This claim is furtherweakened by neurophysiological and clinicalevidence from brain injuries in which failuresin the perception of objects or their (real)motions were accompanied by correspondingfailures in the imagination of those objects(Bisiach & Luzzatti, 1978) or by the experi-ence (of the type to be considered) of theirapparent motions (Zihl, von Cramon, & Mai,1983).

Endogenous Biological Rhythms as a Modelfor Internal Representation

Because the circadian behavioral cycle iscorrelated with the presence or absence ofdaylight, people long drew the inference thatan animal's emergence from and return toits nest or burrow was wholly controlled by


this obvious external stimulus. It was littlemore than 50 years ago, when experimentersfirst began to maintain animals in artificiallaboratory conditions of constant illuminationand temperature, that they discovered thatthe circadian (and even the circannual)rhythm had in fact been internalized (Biin-ning, 1973). Hamsters, for example, wouldcontinue their cycles of alternating activityand sleep indefinitely in the absence ofa corresponding environmental periodicity,each animal maintaining a cycle of 24 hoursplus or minus no more than a few minutesper day (see Rusak & Zucker, 1975).

Of course, a few minutes of deviation froma 24-hour cycle in each animal would causeit gradually to drift out of phase with otheranimals in the laboratory and with the truediurnal cycle. Yet, no more than a briefperiod of increased illumination introducedat the same time each day, or even at thesame time just on occasional days, wouldentrain the endogenous cycles and resyn-chronize all the animals in the laboratory.

Here is an environmental regularity thathas continued with celestial-mechanical pre-cision throughout biological evolution. Eventhough it is correlated with the waxing andwaning of daylight, this periodicity has be-come internalized so that it continues auton-omously in the absence of the correlatedstimulus, freeing the animal from a directdependence on that stimulus. Thus, a diurnalanimal while still in the darkness of its burrowcan begin to awake and to prepare for activeemergence toward the onset of sunrise, andcan do so as well on a cloudy as on a sunnyday. At the same time, the animal can usewhat photic cues (weak or strong) are availableas to the true onset and offset of daytime tokeep its internal cycle in synchronous tuning.

Perception is very much like this. Underfavorable conditions of illumination, mobility,and so on, our experience of the environmentis so tightly guided by the externally availableinformation that we readily feel the appro-priateness of Gibson's term direct perception(J. Gibson, 1972; also see Austin, 1962; Mi-chaels & Carello, 1981). At the same time,however, we know that our perceptual expe-rience is mediated by many complex thoughhighly automatic neural processes. Any in-terruption of these processes by drugs, acci-

dent, or disease can alter or disrupt percep-tion. Moreover, these processes embody con-straints appropriate only to the world inwhich we have evolved. Therefore, just as ananimal that had evolved on a planet with avery different period of rotation would notsynchronize well to our daily cycle, a beingthat had evolved in a radically different worldwould not perceive this one in the way thatwe do—even under favorable conditions. Pre-cisely because our own internal constraintsso well match the external constraints in ourworld, these internalized constraints revealthemselves only when externally availableinformation is degraded or eliminated. Beingless tightly controlled from without, activityin the perceptual system is then necessarilyguided more by whatever constraints operatewithin.

Internalized Constraints ofKinematic Geometry

I believe the external constraints that havebeen most invariant throughout evolutionhave become most deeply internalized, as inthe case of the circadian rhythm. Such con-straints may be extremely general and ab-stract: The world is spatially three dimen-sional, locally Euclidean, and isotropic exceptfor a gravitationally conferred unique uprightdirection, and it is temporally one dimen-sional and isotropic except for a thermody-namically conferred unique forward direction(see Davies, 1977). In it, material bodies arebounded by two-dimensional surfaces andmove, relative to each other, in ways that canbe approximately characterized, locally andat each moment, by six degrees of freedom(three of translation and three of rotation).Light, until absorbed or deflected by thesurface of such bodies, travels between themin straight lines and at a constant, vastlygreater velocity. Consequently, the optical in-formation about other bodies available at thesensory surface of each organism is governedby the geometrical laws of perspective projec-tion.

The constraints with which I am primarilyconcerned are those of kinematic geometry(Hunt, 1978, p. 2), which govern the relativemotions of rigid objects, or of local parts ofnonrigid objects, during brief moments of


time. Although there are infinitely many waysin which an object might be moved from anyposition A to any other position B, in three-dimensional space there is a simplest way ofeffecting the displacement—a fact that wasestablished between 1763 and 1830 throughthe efforts of Mozzi, Giorgini, and finally,Chasles (1830; see Ball, 1900, pp. 4, 510;Hunt, 1978, p. 49). For any two positions, Aand B, Chasles's theorem states that there isa unique axis in space such that the objectcan be moved from A to B by a rotationabout that axis together with a simultaneoustranslation along that same axis: a helicaltwist or "screw displacement" (Ball, 1900;Coxeter, 1961; Greenwood, 1965). Moreover,even for an arbitrary motion between A andB, the motion at any instant in time willapproximate a twisting of this kind about amomentarily unique axis (Ball, 1900, p. 10).A twist thus bears the same relation to arigid body as an ordinary vector bears to apoint, the special cases of pure rotation andpure translation being realized as the pitchof the twist becomes zero or infinite, respec-tively.

I consider also the two-dimensional caseof Chasles's theorem: For any two positions,A and B, of a two-dimensional object in theplane, there is always a unique pivot point,P, such that the object can be displaced fromA to B by a rigid rotation in the plane aboutP (Coxeter, 1961). Here, pure translation isrealized as the pivot point P recedes to thepoint at infinity in a direction orthogonal tothe direction of the translational displacement.As before, an arbitrary motion between Aand B will, at any instant t, approximate arigid rotation about a momentarily uniquepoint, P(f).2

Illustrative Experiments onApparent Motion

The phenomenon of apparent motion,which seems to fall somewhere between per-ception and imagery, provides perhaps thebest illustration of how internalized con-straints of kinematic geometry may governthe perceptual/imaginal representation of ob-jects and their transformations. In apparentmotion, the alternating presentation of twodifferent views of an object gives rise to the

experience of one object smoothly transform-ing back and forth—provided both that thetime between the onset of one view and theonset of the other (called the stimulus onsetasynchrony, SOA) is not too short and thatthe time between the offset of one view andthe onset of the other (called the interstimulusinterval, ISI) is not too long. That thesetransformations are experienced as traversingwell-defined trajectories is of the greatestsignificance: In the absence of any externalsupport for such trajectories, the form theytake provides an indication of what I call theinternalized constraints.

Internalized Constraints Revealed inApparent Motion

In what is perhaps the simplest case ofapparent motion, already investigated byHelmholtz's student Exner (1875) and thenby the founder of Gestalt psychology, Wert-heimer (1912), two laterally separated dotsare presented in alternation. For appropriatetime intervals, the experience is of a singledot moving back and forth over the straightpath between the two positions of presenta-tion. We thus have an intimation that theexperienced impletion is an embodiment ofgeneral principles of object conservation andleast action (Shepard, 1981b). The richnessof these internalized principles is revealed inrecent experiments in which the two alter-nately presented stimuli are views of morecomplex objects differing by more complextransformations—transformations of (in ad-dition to translation) rotation, reflection, ex-pansion or contraction, and various combi-nations of these. (See Bundesen, Larsen, &Farrell, 1983; Farrell, 1983; Farrell, Larsen,& Bundesen, 1982; Farrell & Shepard, 1981;Foster, 1975; Shepard & Judd, 1976.)

In Figure 2, each of the 12 panels shows adifferent pair of views of a polygonal object,

2 In three dimensions, the displacement of a point hasthree degrees of freedom (two for the direction of thecorresponding vector and one for its magnitude) and thedisplacement of a rigid object has six (four for the axisof the twist and the fifth and sixth for its pitch andamplitude). Similarly, in two dimensions, the displace-ments of a point and of a rigid object have, respectively,two and three degrees of freedom.


a. Translation, T

Translation In frontal plane

Oblique translation in depth

g. Mirror reflection, M

180° Rotation In depth

T * S + R

Oblique Screw Displacement

b. Size scaling, S

Translation in depth

Screw displacement in depth

h. Affine contraction, A

60 Rotation in depth

k. T + R * M

Oblique Screw Displacement

c. Rotation, R

Rotation in frontal plane

f. T * R

Rotation in frontal plane

f. Affine contraction, A*

90 Rotation in depth

I. T t R + A

Oblique screw displacement

Figure 2. Pairs of two-dimensional shapes that when alternately presented in the indicated positions withinthe same (circular) field, give rise to rigid apparent motion in space. (For each pair, the transformationthat maps one shape into the other has the form indicated above the pair if the transformation is confinedto the picture plane, and the form indicated below if it is the simplest rigid transformation in space.)

which might be displayed in alternation withinthe same two-dimensional field. Thus, Panela depicts the case in which the polygonalternately appears on the left and the rightof a circular field, giving rise to back-and-forth apparent motion. The polygon is oneof the forms of the type introduced by Att-neave & Arnoult (1956) that Lynn Coopergenerated and used to such advantage in herelegant series of experiments on mental ro-tation (Cooper, 1975, 1976; Cooper & Pod-gorny, 1976) and that Sherryl Judd and Ilater adopted for some of our investigationsof apparent rotational motion (see Shepard& Cooper, 1982, p. 313).

As indicated at the top of the panels, eachpair illustrates a way in which the two viewsmight be related by a transformation in thepicture plane: in the top row, shape-preservingtransformations of translation (T), size scaling(S), and rotation (R); in the second row,combinations of two of these shape-preservingtransformations (T + S, S + R, and T + R);

in the third row, shape-altering affine3 trans-formations (A, its degenerate case A*, andits negative extension or mirror reflection M);and in the last row, combinations of threetransformations (T + S + R, T + R + M,and T + R + A). When they are thus definedas transformations within the plane of thepicture, only in 3 of the 12 pairs are the twoviews related by a rigid motion of the planarpolygon: those in Panels a, c, and f, whichare composed only of translations, rotations,or both. In each of the nine remaining pairs,the transformation within the plane is non-rigid because it includes a change in thepolygon's size, shape, or in both the size andshape.

Nevertheless, in each of these cases, if therate of alternation is not too great, the motiontends to be experienced as the rigid transfor-

3 An affine transformation permits differential linearexpansion or contraction along different directions butpreserves straightness and parallelism of lines.


mation prescribed by Chasles's theorem, asindicated below each pair. Invariance in per-ceived size and shape is achieved by liberatingthe transformation and the object from theconfines of the picture plane into three-dimensional space. Thus a viewer tends toexperience for Panel b an approach andrecession rather than an expansion and con-traction; for Panel e a unified twisting ap-proach and recession (the helical or screwlikemotion) rather than a rotation, expansion,and contraction; and for Panels h and i,respectively, a 60° or 90° rotational oscillationabout a vertical axis, rather than a horizontalcompression and expansion.4

Out of the infinite set of transformationalpaths through which the one shape could berigidly moved into congruence with the other,one tends to experience that unique, mini-mum twisting motion prescribed by kinematicgeometry. The axis of the helical motion mayhowever be aligned with the line of sight (asin Panels c, e, f), orthogonal to the line ofsight (as in Panels g, h, i), or oblique (as inPanels j, k, 1), and the pitch of the: twist maybe zero, yielding purely circular motion (asin Panels c and f), or it may become infinite,yielding purely translational motion, whetherit is one that is confined to the plane (as inPanel a), orthogonal to the plane (as in Panelb), or oblique (as in Panel d).

Abstractness of InternalizedPerceptual Constraints

The two-dimensional case of Chasles'stheorem provides the simplest illustration ofthe abstractness of the internalized con-straints. From considerations of physical dy-namics, one might guess that two planarfigures alternately presented in positions thatdiffer arbitrarily (and hence by both a trans-lation and a rotation, as in Panel f of Figure2) would give rise to an apparent motion inwhich the center of mass of the apparentlymoving body traverses the shortest, straightline between its two terminal positions. Be-cause the two views also differ by a rotation,such a motion would have to be accompaniedby an additional, apparent rotational trans-formation, as illustrated for two rectangles inFigure 3, Part a. Instead of such a doubletransformation, however, Foster (1975) found

Figure 3. Intermediate positions of a rectangle (drawn inthin lines) between the same two rectangles (drawn inheavy lines), which differ arbitrarily in both position andorientation, along a path consisting of a combined recti-linear translation and a rotation (Part a), and the path(which Foster, 1975, found to be preferred in apparentmotion) consisting of a rotation only (Part b). (FromMental Images and Their Transformations by R. N.Shepard and L. A. Cooper, 1982, p. 316. Copyright 1982by The Massachusetts Institute of Technology. Adaptedby permission.)

that the motion is generally experienced overa curved path. By having observers adjust thevariable intermediate rectangle (indicated inFigure 3 by thinner lines) so that it appearedto fall on the path of motion, he found that(under conducive conditions) the motiontended to be experienced over that uniquecircular path that rigidly carries the onefigure into the other by a single rotationabout a fixed point, P, in the plane, as shownin Figure 3, Part b.

It seems that here, as in the case of themoire pattern of Glass (1969; an example ofwhich is shown in Figure 4, Part b), thevisual system picks out the fixed point impliedby the two presented positions of a rigidconfiguration in the plane and, hence, iden-tifies the two configurations with each otherby means of a simple rotation. (See Foster,1975, 1978; Shepard, 1981b; and for a reviewand theoretical discussion, Shepard & Cooper,

4 In an investigation of apparent motion motivated bysimilar objectives, Warren (1977) reported that alternationbetween two-dimensional shapes differing by an affinetransformation did not yield rigid apparent motion.However his allegedly affine pair (Panel g) was not affine,and his instructions and resulting subjective reports areopen to questions of interpretation, choice of criterion,and effects of perceptual set or expectancy.


Figure 4. Moire pattern described by Glass (1969), inwhich two identical transparencies of a random texture(Part a), when superimposed in an arbitrary misalignment,give rise to the appearance of concentric circles (Part b).(As one transparency is shifted with respect to the other,the center of the concentric circles moves in an orthogonaldirection.)

1982.) Incidentally, the visual system alsoextracts fixed points in the case of nonrigidtransformations, as has been demonstratedby Johansson (1950, 1973), Wallach (1965/1976), and most extensively by Cutting andhis associates (see Cutting, 1981; Cutting &Proffitt, 1982).

There are good reasons why the automaticoperations of the perceptual system shouldbe guided more by general principles of ki-nematic geometry than by specific principlesgoverning the different probable behaviors ofparticular objects. Chasles's theorem con-strains the motion of each semirigid part ofa body, during each moment of time, to asimple, six-degrees-of-freedom twisting mo-tion, including the limiting cases of purerotations or translations. By contrast, themore protracted motions of particular objects(a falling leaf, floating stick, diving bird, orpouncing cat) have vastly more degrees offreedom that respond quite differently tomany unknowable factors (breezes, currents,memories, or intentions). Moreover, relativeto a rapidly moving observer, the spatialtransformations of even nonrigid, insubstan-tial, or transient objects (snakes, bushes,waves, clouds, or wisps of smoke) behave likethe transformations of rigid objects (Shepard& Cooper, 1982).

It is not surprising then that the automaticperceptual impletion that is revealed in ap-parent motion does not attempt either theimpossible prediction or the arbitrary selec-tion of one natural motion out of the manyappropriate to the particular object. Rather,it simply instantiates the continuing existenceof the object by means of the unique, simplestrigid motion that will carry the one view intothe other, and it does so in a way that iscompatible with a movement either of theobserver or of the object observed.

Possibly some pervasive principles of phys-ical dynamics (such as a principle of momen-tum), in addition to the more abstract prin-ciples of purely kinematic geometry, havebeen internalized to the extent that theyinfluence apparent motion (Foster & Gravano,1982; Freyd, 1983a,. 1983c, 1983d, 1983e;Freyd & Finke, 1984; Ramachandran & An-stis, 1983). But there evidently is little or noeffect of the particular object presented. Themotion we involuntarily experience when apicture of an object is presented first in oneplace and then in another, whether the pictureis of a leaf or of a cat, is neither a flutteringdrift nor a pounce; it is, in both cases, thesame simplest, rigid displacement. True, wemay imagine a leaf fluttering down or a catpouncing, but in doing so we voluntarilyundertake a more complex simulation (justas we might in imagining a leaf pouncing ora cat fluttering down). Such mental simula-tions may be guided by internalizations ofmore specific principles of physical dynamicsand even perhaps of animal behavior.

Pervasive Constraints of Time and Distance

I have taken the sources of the perceptualconstraints considered so far to be corre-sponding constraints in the world, for exam-ple, the 24-hour diurnal cycle and principlesof kinematic geometry and perhaps of phys-ical dynamics. However, there are other highlyorderly perceptual regularities that may notbe reflections of constraints that happened toprevail in our world so much as manifesta-tions of constraints that are unavoidable inany system that could exist in this world.Thus, much as the velocity of light limits thespeed of communication between distantbodies, the necessarily finite velocity of signal


"3 180r

.9 160 -

140

; 120

2 100

a. Corbin (1942)Departure from frontal plane:• 0 degrees• 60 degrees

„ 350,

250'

800

200,

O 100

b- Shopard&Judd (1976)

• Picture-plane rotations• Depth rotations

2 4 6 8 1 0Physical Separation (Inches)

60 100 140Difference in Orientations (degrees)

180

Figure 5. Minimum stimulus-onset asynchronies (critical SOAs) for good apparent motion as a functionof extent of transformation in three-dimensional space, as obtained by Corbin (1942) for translationalmotion (Part a) and by Shepard and Judd (1976) for rotation (Part b). (Note, in both cases, the linearityof the data and the similarity in slope between the data for transformations parallel to the frontal planeand for transformations in depth. Part a is from Mental Images and Their Transformations by R. N.Shepard and L. A. Cooper, 1982, p. 306. Copyright 1982 by The Massachusettes Institute of Technology.Adapted by permission. Part b is from "Perceptual Illusion of Rotation of Three-Dimensional Objects"by R. N. Shepard and S. A. Judd, 1976, Science, 191, p. 953. Copyright 1976 by the American Associationfor the Advancement of Science. Adapted by permission.

propagation within a body must limit itsprocessing of information (perhaps with con-sequences analogous to those of special rela-tivity—cf. Caelli, Hoffman, & Lindman,1978). Therefore, the possibility of a simplerigid transformation between two alternatelypresented views is not alone sufficient for thebrain to instantiate that transformation as arigid apparent motion. The extent of thetransformation must not be too great inrelation to the time available for its neuralimpletion. Similarly, in connection with theexperiment by Foster (1975), the distance tothe center of rotation and/or the angle of thatrotation must not be too large (cf. Farrell,1983; Mori, 1982).

In line with these expectations, the mini-mum SOA that yields apparent motion overa particular path generally increases linearlywith the length of that transformational path.In the case of simple translational apparentmotion, such a relation was enunciated asthe third law of apparent motion by Korte(1915). However, a linear relation of this kindholds for other types of transformations aswell, including rotations (Shepard & Judd,1976), expansions or contractions, and com-binations of these with rotations and trans-lations (Bundesen, Larsen, & Farrell, 1983;Farrell, 1983; Farrell et al. 1982). We have

also found such a relation for apparent motionover curved paths externally defined by flash-ing, very briefly and at low contrast, a partic-ular path during the interstimulus interval(Shepard & Zare, 1983).

These critical times have confirmed thatwhat is being represented (in the absence ofreal motion) is a transformation of the distalobject in three-dimensional space and not atransformation of its projection on the retina(Attneave & Block, 1973; Corbin, 1942; Oga-sawara, 1936; Shepard & Judd, 1976). Figure5, Parts a and b, shows the closeness of theagreement between the critical times for ap-parent motion in the picture plane and indepth for translational apparent motion (Cor-bin, 1942) and for rotational apparent motion(Shepard & Judd, 1976).

The phenomena of apparent motion arisein the auditory and in the tactual modalitiesas well (see, e.g., Kirman, 1983). Moreover,the linear dependence of critical time ontransformational distance has been found evenwhen the transformation is not literally spa-tial. For example, there is a similar increasein critical SOA with increasing separation inpitch between two alternately presented tones(see Jones, 1976; McAdams & Bregman,1979; Shepard, 198 Ib, 1982a; van Noorden,1975).


Phenomenally Distinct Modesof Apparent Motion

Some pairs of stimuli can be transformedinto each other by different transformationsof approximately equal extent. For example,if the two alternately presented orientationsof an asymmetric object differ by 180°, therotational apparent motion can be experi-enced in either direction through equal angles(Farrell & Shepard, 1981; Robins & Shepard,1977; Shepard & Judd, 1976). An analogousambiguity occurs in auditory pitch. I haveargued (Shepard, 1982a) that Chasles's theo-rem similarly constrains the motions of rigidauditory objects (e.g., melodies and chords)in pitch space. Because pitch possesses cir-cular components, one can synthesize tonesthat differ only in their orientations arounda chroma circle (Shepard, 1964). As a con-sequence, when two tones that are diametri-cally opposite on this circle are sounded inalternation, they are heard as moving (througha tritone interval of pitch) in either of twoways (up-down-up-down- . . . or down-up-down-up- . . .) corresponding to oppositedirections of movement around the chromacircle (see Shepard, 1983, and hear the ac-companying Sound Demonstration 4).

In both visual and auditory cases, theapparent motion experienced can depend onthe rate of switching between stimuli (Farrell& Shepard, 1981; Shepard, 1981b; Shepard& Zare, 1983). For example, we have repli-cated Brown and Voth's (1937) finding thatwhen dots are cyclically flashed at the fourcorners of a square, the apparent motionfollows the straight paths between successivecorners for slow rates of switching but be-comes a continuous circular motion at higherrates. Here too, under conducive conditions,a fixed point is evidently extracted, permittingthe representation of a single transformation(a continuous rigid rotation about that fixedpoint) in place of four successive transfor-mations (e.g., linear translations repeatingthrough the cycle: move right, down, left, up,. . .). The conducive conditions in this casepresumably require that the time withinwhich three successive dots appear (the min-imum number necessary to define the centerof the circle) fall within the relevant perceptualintegration time.5

Figure 6. Alternately presented halves of a low-contrasthomogeneous elliptical path (Panels a and b), and ex-amples of particular modes of the two principal types ofapparent motion experienced: a circular rim spinningabout a vertical axis (Panel c) and a "jump rope" whirlingabout a horizontal axis (Panel d).

Even in the simplest case of the path-guided apparent motion studied by Shepardand Zare (1983)—namely, that in which thefaint path that is briefly flashed between thetwo alternately presented dots is the shortest,straight path—the usual report of a recipro-cating or back-and-forth motion of a dot isoften replaced, at higher rates of alternation,by reports of a rapidly spinning disk viewededge on or, occasionally, of a horizontal rodrapidly spinning about its own axis. Followingup these observations, Susan Zare and I havebeen systematically investigating a display inwhich the upper and lower halves of a low-contrast elliptical path are briefly displayedin alternation (Figure 6, Panels a and b). Thisdisplay gives rise to a variety of alternativepercepts.

At high rates of alternation (between SOAsof 50 and 100 ms), observers most oftenexperience a circular rim spinning in a planetipped back in depth (Figure 6, Panel c), anddo so in one of four modes corresponding towhether the plane is experienced as viewedfrom above or below and whether the spinningmotion in that plane is experienced as clock-

5 Similarly, I suggest that a seemingly related phenom-enon of apparent motion reported by Ramachandranand Anstis (1982), though interpreted by them in termsof a dynamical principle of visual momentum, could justas well be interpreted, in terms of the more abstractprinciples of kinematic geometry advanced here, as theextraction of a globally simpler, overall rectilinear motion.


wise or counterclockwise. These are variantsof path-guided apparent motion (Shepard &Zare, 1983) in that the motion is experiencedalong the presented curve. At slower rates(beyond SOAs of 100 ms), observers moreoften experience a "jump rope" whirlingaround a horizontal axis (Figure 6, Panel d)and do so in one of several modes corre-sponding to whether the rope goes down infront and up in back, whirls in the oppositedirection, or oscillates up and down. Theseare variants of standard apparent motion inthat it is the presented stimulus that is expe-rienced as moving—along a path that is notitself presented. Other percepts may also arise;at relatively fast rates, these include what aredescribed as jaws or a clam shell vibratingbetween open and partially closed and, atslower rates, something whirling around theperimeter of a disk that is at the same timewobbling up and down. Occasionally, a sec-ond-harmonic variation of the jump rope isdescribed, in which one side of the ropeappears to go up while the other goes downand then vice versa (yielding a horizontallyoriented figure-eight pattern of oscillation).These various preferred modes of experiencedimpletion may reflect what are, in each case,the simplest motions in three-dimensionalEuclidean space for which the distances ofmotion are compatible with the time allowedfor the internal impletion of such a motion(the SOA).

A Competence-Performance Distinctionfor Perception

The pairs illustrated in Figure 2 generallyinduce an experience of a transformation inthree-dimensional space because only in thisway can the size and shape of the objectuniformly be represented as invariant. Like-wise, the transformations experienced for thepairs shown in Figure 2, Part f, and in Figure3 consist of a rotation about a point in theplane exterior to the object, rather than abouta point that is interior to the object but thatalso undergoes a translation, because only inthis way is the transformation represented aspivoting around an invariant point. Thismuch is harmonious with Gibson's emphasison invariance. However, unlike Gibson, Ihave sought quantitative determinations of

exactly when the ability of the perceptualsystem to capture an invariant breaks down,as an experimentally controlled display de-parts more and more from conditions thatare conducive for the capture of that invari-ance.

Gibson did not concern himself with fail-ures to achieve (or to extract) invariancebecause he confined himself to the mostconducive conditions. Instead of investigatingapparent motion, he studied real motion. Atthe other extreme, many vision researchers,who often presented only extremely impov-erished and nonconducive stimuli, havetended to undervalue the capacity of theperceptual system to represent invariances ofa high order. I suggested (see Shepard, 1982a)that the two approaches might be reconciledby applying to the study of perception,the competence-performance distinction thatChomsky (1965) proposed for the study oflanguage. Information-processing limitationsthat prevent people from producing or com-prehending certain very long sentences donot preclude that people normally produceand comprehend shorter sentences by meansof internalized rules of syntax. Similarly,information-processing limitations that pre-vent people from computing the rigid trans-formation between two very widely separatedviews of an object do not preclude that theynormally compute such a transformation be-tween less widely separated view by meansof internalized principles of kinematic ge-ometry.

Some Relations to Past and Future Studiesof the Representation of Motion

There has of course been a considerablehistory of investigations into the role of rigid-ity in the perception of motion (e.g., Ames,1951; Braunstein, 1976; Dunker, 1929/1937;E. Gibson et al., 1978; Johansson, 1950,1973, 1975; Metzger, 1953; Profntt & Cutting,1979; Restle, 1979; Spelke, 1982; Wallach,1965; Wallach & O'Connell, 1953) and ofapparent motion (e.g., Foster, 1972; Hochberg& Brooks, 1974; Kolers, 1972; Kolers &Pomerantz, 1971; Mori, 1982; Navon, 1976;Orlansky, 1940; Squires, 1959; Warren, 1977).The results are generally consonant with thenotion that the perceptual system tends to


represent a motion as rigid under conduciveconditions. However, in the absence of aunified framework for specifying which par-ticular rigid motion is chosen and for char-acterizing the conducive conditions, specificconclusions have varied from one study ofapparent motion to another.

With regard to the selection of a particularmotion, I proposed that out of the infiniteset of possible rigid motions, an observertends to experience the simplest helical mo-tion (including its limiting circular or recti-linear motions) prescribed for three-dimen-sional Euclidean space by kinematic geometryand, specifically, by Chasles's theorem. Incase there are alternative motions of this typethat are equal or nearly equal in extent (suchas 180° rotations in opposite directions), Iclaim that observers experience only one ofthese motions on any one trial but that theycan be predisposed towards a particular oneof these motions by presenting, for example,a corresponding real motion just before thetrial. By implication, I also claim that motionsthat are not of this simplest helical type,whether rigid or nonrigid, will not be expe-rienced unless they are forced on the observerby external conditions. Thus, one can devisea sequence of stationary views that will inducethe appearance of, say, a cat pouncing, (ratherthan rigidly translating), but only if one pre-sents (a) beginning and ending views that aredifferent, (b) other intermediate views (as instroboscopically or cinematically displayedanimation), or (c) the blurred path of motion(as described by Shepard & Zare, 1983).

With regard to the conducive conditionsfor impletion of a particular apparent motion,I have proposed two primary requirements:(a) The ISI between sequentially presentedviews must fall within the appropriate periodof temporal integration, (b) Correspondingparts of successive views must fall within theappropriate range of spatial integration rela-tive to the SOA available for making theconnections, and relative to the prevalence ofsimilar but noncorresponding parts. Onlythen can the observer identify correspondingparts of the two views and complete theglobal transformation that rigidly carries thosein one view into the corresponding ones inthe other view (Attneave, 1974; Farrell &Shepard, 1981; Shepard, 1981b; Ullman,

1979). More specifically, as a generalizationof Korte's third law, I have claimed that inthe absence of strongly competing alternativetransformations, the critical SOA (i.e., theminimum time between stimulus onsetsneeded to complete this rigid transformation)increases linearly with the extent of the trans-formation, whether that transformation isrectilinear (Corbin, 1942; Korte, 1915), cir-cular (Shepard & Judd, 1976; Shepard &Zare, 1983), or helical (Shepard, 1981b).

Putting the considerations concerning pref-erence for the simplest transformation thatpreserves rigid structure together with thoseconcerning the conducive conditions for im-pletion of such a transformation, I have pos-ited a hierarchy of structural invariance(Shepard, 1981b). At the top of the hierarchyare those transformations that preserve rigidstructure but that require greater time fortheir impletion. As the perceptual system isgiven less time (by decreasing the SOA), thesystem will continue to identify the two viewsand hence to maintain object conservation,but only by accepting weaker criteria forobject identity. Shorter paths that short-circuitthe helical trajectory will then be traversed,giving rise to increasing degrees of experiencednonrigidity (Farrell & Shepard, 1981). Like-wise, if the two alternately presented viewsare incompatible with a rigid transformationin three-dimensional space, the two viewswill still be interpreted as a persisting object,but again a nonrigid one.

These considerations provide a basis forreconciling many of the apparent inconsis-tencies in the literature on rigid apparentmotion. Often, experiments that (a) fail toobtain rigid motion between two views of thesame object or (b) fail to obtain the simplestmotion prescribed by Chasles's theorem havenot ensured that the SOA was sufficientlylong (when the transformations were large)and/or that the observers were sufficientlyprimed for that particular motion (when thecompeting alternatives were strong).

The theory outlined leads to a number ofexpectations that remain to be empiricallytested. The simplest helical motion that dis-places an asymmetric object from one posi-tion to another is generally unique, exceptfor cases in which there are equivalent alter-native paths (e.g., 180° rotations in opposite


directions). However, there are always otherhelical motions that yield the same result,but by means of a larger number of rotations.Moreover, in the case of symmetrical objectsthere are still more possibilities. Thus, ahorizontal rectangle alternately displayed onthe left and right could be seen as translatingback and forth, rotating through 180° in thepicture plane (either above or below), rotating180° in depth (either in front or in back),and so on.

All such transformations correspond togeodesic or locally shortest paths in the curvedmanifold of distinguishably different positionsof the object, that is, to the analogues ofstraight lines in Euclidean space, great circleson the surface of a sphere, or helices on thesurface of a torus (Shepard, 1978a, 198 Ib).Accordingly, I predict that when alternativegeodesic paths are not too widely different inlength, observers can be induced (e.g., by apreceding display) to experience transforma-tions over different ones of these alternativepaths, with critical SOAs proportional to thelength of each path. I further predict thatmotions cannot be induced in this way alongarbitrary paths that are not geodesic, andthat the semantic interpretation of the objectwill in any case have little or no influence onthe path of motion or its critical SOA.

Determinants of Internal Representations

The fact that the same alternating visualor auditory display can lead to distinctlydifferent apparent motions reinforces thepoint, often made on the basis of otherambiguous stimuli (such as the Necker Cube),that perception cannot adequately be de-scribed simply as an individual act of pickingup an invariant that is present in that partic-ular stimulus. What is perceived is determinedas well by much more general and abstractinvariants that have instead been picked upgenetically over an enormous history of evo-lutionary internalization. Although someconstraints (e.g., of the sorts considered byChomsky, 1965;Freyd, 1983b; or Keil, 1981)may not have an external origin, I find suchan alternative to be less appealing because itwould seem to imply that those constraintsare arbitrary (cf. Shepard, 1981b, 1982b).6

Accordingly, I propose a tentative classifica-

tion of the determinants of internal represen-tations into immediate external determinantsand three subclasses of internalizations oforiginally external determinants.

Immediate External Determinants

Here I include all (variant and invariant)information that is available in the opticarray and in the corresponding arrays of theother senses of hearing, touch, and so on,within what I have been calling the relevant"period of temporal integration."

Internal Determinants

I classify any determinants that do not fallunder immediate external determinants asinternal because they are, by this rule ofclassification, not externally acting on theorganism within the given period of temporalintegration. However, these determinants aremostly internalizations of current or previ-ously prevailing external circumstances—al-though of increasingly remote origin as spec-ified:

I. Determinants temporarily established bythe current context. Here I include both (a)transitory bodily or emotional states (whichare, in turn, largely determined by precedingexternal circumstances, such as presence orabsence of food or traumatic events) and (b)mental sets or attentional biases (which arelargely established by the external context,including such things as preceding stimuliand instructions given in a psychological ex-periment). For example, we can predisposean observer toward either of two alternativeapparent motions by presenting the corre-sponding real motion just before (see Shepard,1981b; Shepard & Cooper, 1982). Analo-

61 conjecture that the elaborate, special apparatus ofsyntax has evolved in humans primarily for one purpose:to furnish automatic rules for mapping between complex,multidimensional structures in the representational systemand one-dimensional strings of discrete communicativegestures (vocal or manual). I have also argued, however,that these rules, which could not have sprung full fledgedfrom nowhere, may have been built upon already highlyevolved rules of spatial representation and transformation(Shepard, 1975, 198 Ib, 1982b). If so, syntactic rules maybe to some extent traceable, after all, to abstract propertiesof the external world.


gously, a sequence of two tones on oppositesides of the computer-generated chroma circlewill be heard as jumping up or jumpingdown in pitch when immediately precededby an unambiguously rising or falling se-quence, respectively (Shepard, 1983, SoundDemonstration 4).

2. Determinants acquired through past ex-perience by each individual. These are themore enduring but modifiable constraintsthat have been internalized through learningor perceptual differentiation (E. Gibson, 1969;J. Gibson & E. Gibson, 1955). For example,perceptual discrimination is better (a) in thecase of adults, between upright than betweeninverted faces (e.g., Carey, 1981; Hochberg& Galper, 1967; Yin, 1969), and (b) in thecase of chess masters, between board positionsthat might occur in an actual game of chessthan between ones arranged at random (e.g.,Chase & Simon, 1973; de Groot, 1965).

3. Determinants incorporated into the ge-netic code during the evolution of the species.These place constraints on each individualthat are predetermined at the time of birth.Because the internalization of these con-straints has taken place over by far the longestspan of time, they presumably tend to reflectthe most enduring and ubiquitous invariancesin the world. I have conjectured (Shepard,1981b) that they include those that enable usto perceive a rigid rotation (or, generally,helical motion) on the basis of a two-dimen-sional projection of a moving three-dimen-sional structure (Wallach & O'Connell, 1953;also see Braunstein, 1976; Green, 1961; Noll,1965), and to do so from early infancy (E.Gibson et al., 1978; Spelke, 1982), but leaveus unable to perceive rigid motion on thebasis of a similar projection of a movingfour-dimensional structure (whether that pro-jection is two-dimensional, as in a computer-generated film produced by Bert Green, orthree-dimensional, as in a stereoscopic displaylater devised by Mike Noll).

After some delay, of course, a stimulus thatwas an immediate external determinant mustbecome a preceding context and hence aninternal determinant; that is, beyond a certaintemporal integration time, what was a perceptmust shade off into a memory. Likewise,there may be a continuum between a short-term memory (as in Determinant 1) and a

long-term memory (as in Determinant 2).Moreover, some long-term determinants, al-though learned (as in Determinant 2), maybe acquirable only during a critical period ofearly development of the individual (Hess,1959; Lorenz, 1935) and may thereafter re-main as unalterable as one that is geneticallyencoded (as in Determinant 3). Possibly, hu-mans acquire absolute pitch only in this way(Jeffress, 1962). In any case, I assume thatdeterminants of each of the types that I havelisted constrain the determinants of all pre-viously listed types. Thus, genetic endowmentconstrains what can be learned, hence whatcan be attended to, and thence what will beperceived. If so, the actual extraction of in-variants from the externally available infor-mation classified under immediate externaldeterminants is made possible by our biolog-ically internalized constraints. Certainly nei-ther an empty black box nor a randomlywired system can be expected to carry outsuch extractions.

The adaptive significances of all four ofthe listed types of determinants seem clear.An organism must be perceptually responsive(as under immediate external determinants)to the immediate, locally unfolding events,which (even in a deterministic world) couldnever be fully deduced or anticipated (seeFord, 1983). In addition, the organism canprofit by more or less temporarily and flexiblyinternalizing (through contextual guidance orthrough learning) those predictabilities thatare likely to prevail in the immediate situationor throughout the current epoch or locale.Finally, there would be an advantage in havingthe most permanent and certain constraintsin its world prewired (as in Determinant 3);then each separate animal need not run therisks of having to learn those constraints denovo through its own trial and possibly fatalerror. Such prewired constraints would con-stitute internalizations of external constraintsin the very real sense that a being that hadevolved in a very different world would havecorrespondingly different internalized con-straints.

Internal Representation asa Resonance Phenomenon

The closest Gibson came to speaking ofinternal mechanisms subserving perception


was when he likened perception to the phys-ical phenomenon of resonance (Gibson, 1966).Despite the reservations that Gibson (p. 271)himself expressed, I believe that the metaphorof resonance, also proposed for cognition byDunker (1945), alone enables me to makethe main points I wish to make about internalrepresentations and their constraints.

Instead of saying that an organism picksup the invariant affordances that are .whollypresent in the sensory arrays, I propose thatas a result of biological evolution and indi-vidual learning, the organism is, at any givenmoment, tuned to resonate to the incomingpatterns that correspond to the invariantsthat are significant for it (Shepard, 1981b).Up to this point I have not departed signifi-cantly from what Gibson himself might havesaid. Moreover, with the notion of selectivetuning I can encompass the notion of afford-once and thus explain how different organ-isms, with their different needs, pick updifferent invariances in the world.

However, as I pursue the resonance meta-phor further, implications come to light thatare at variance with the prevailing ecologicalapproach. Indeed it may have been this po-tential discord that deterred Gibson from useof the resonance metaphor in his last book(J. Gibson, 1979). However, these furtherimplications seem to be just what is neededto accommodate remembering, imagining,planning, and thinking.

Properties of a Resonant System

The first implication of the metaphor isthat a tuned resonator embodies constraints.Resonators respond differently to the samestimuli, depending on their tuning. The sec-ond implication is that a resonant systemcan be excited in different ways. Most effi-ciently, of course, it is excited by the patternof energy to which it is tuned. (Indeed, itcontinues to ring for a while following thecessation of that stimulus, manifesting a kindof short-term memory.) However, it is alsoexcited, though to a lesser degree, by a signalthat is slightly different, weaker, or incomplete.Finally, it can also be caused to ring quiteautonomously by administering an unstruc-tured impulse from within. An undampedpiano string tuned to middle C (262 Hz)

resonates most fully to a continuing acousticsignal of that particular frequency. But it alsoresonates to some extent to a related acousticsignal that is very brief, is of a slightlydifferent frequency, or stands in some har-monic relation to that frequency. Finally, itsimilarly responds simply to a single blow ofthe padded hammer inside the piano. Thethird implication is that a resonant systemmay have many different modes of excitation.Thus, different disturbances that induce sym-pathetic vibrations in that same middle-Cstring may excite the fundamental and itsvarious harmonics to different relative degrees.

Perhaps the perceptual system has evolvedresonant modes that mirror the significantobjects and their transformations. Whenstimulated by a strong natural signal, asunder favorable conditions of motion andillumination, the system's resonant couplingwith the world would be tight enough to giverise to what Gibson called direct perception.However, the coupling is tight only becausean appropriate match has evolved betweenthe externally available information and theinternalized constraints—just as animals be-haviorally resonate to illumination brieflyintroduced at 24-hour intervals only becausethey have already internalized the 24-hourperiod of the earth's rotation.

Even when there is generally an appropriatematch, the information available in particularsituations may be impoverished, as in a noc-turnal, brief, obstructed, schematic, or pic-torial view. Necessarily, the system is thenless tightly coupled to that information. Theresulting resonant response may neverthelessbe quite complete, as in the many phenomenaof perceptual filling in, subjective contours,amodal completion, and path impletion (inthe various phenomena of apparent motion),but it may also be much less stable and, asin the perception of ambiguous stimuli, mayexhibit different modes of resonance on dif-ferent occasions. Finally, in the completeabsence of external information, the systemcan be excited entirely from within. Some-thing internal may "strike the mind," givingrise to the various "ringings" that we callmental images, hallucinations, and dreams.7

7 The first occasion on which I myself advanced theidea that imagery and dreams correspond to the sponta-


Of course, the piano is an inadequatemodel in several respects. The tendency forone perceptual interpretation to dominate itsalternatives at any one time implies a mech-anism of mutual inhibition that the pianolacks. Also, unlike the different modes ofresonance of a piano string, the differentmodes of resonance in the perceptual systemare not related by anything so rigid as inherentfrequency ratios. Through evolution, learning,and contextually induced states of attention,the resonances of the perceptual system havebeen shaped instead to mesh with the externalworld (Shepard, 1981b).

Hierarchical Organization ofthe Resonant Modes

Even within a piano, a complex acousticevent may simultaneously excite many differ-ent modes of resonance; that is, sympatheticvibrations arise to different degrees at certainharmonics in particular (undamped) strings.Similarly, the perceiving of a complex objector event, such as a rotating cube or a laughingface, presumably corresponds to the excitationof many different resonant modes of theperceptual system. Moreover, these modesvary from those that resonate to very specific,sensory features such as the particular length,direction, and motion of an edge of the cubeor the particular size, color, and texture ofthe iris of an eye, to those that resonate tomore abstract, conceptual categories such asthe presence of rotation (regardless of theobject rotating) or of a face (regardless ofage, sex, color, hairstyle, expression, orienta-tion, or distance). There is therefore reasonto suppose that perceptual processes are inthis sense hierarchical, following neurophys-iologists (e.g., Gross, Rocha-Miranda, &Bender, 1972; Hubel & Wiesel, 1965; Konor-

neous internal excitation of a perceptual system that hasevolved to resonate with natural processes in the externalworld was a meeting of a student-run Monday EveningDiscussion Group at Yale while I was a graduate studentthere in the early 1950s. The idea has, if nothing else,the virtue of not requiring the assumption that duringdreaming, some other part of the brain must, in themanner of a movie projector, play upon the cortex withspecifically programmed patterns of excitation—as seemedto be implied by the otherwise admirable neurophysio-logical account of dreaming offered by Dement (1965).

ski, 1967; Lettvin, Maturana, McCulloch, &Pitts, 1959), computer scientists (e.g., Marr,1982; Selfridge, 1959), experimental psychol-ogists (e.g., Bruner, 1957; Neisser, 1967, p.254; Posner, 1969; Shepard, 1975), and phi-losophers (e.g., Price, 1946; James, 1890/1950, p. 49; cf. also Kant, 1781/1961, pp.104-106, on schemata).

An important qualification, however, isthat one mode is not assigned to a higherlevel than are other modes in the hierarchybecause its excitation is preceded or causedby excitation of those other modes. Rather,it is assigned to the higher level solely becauseit resonates to a wider natural class of externalobjects or events. Thus the mode that repre-sents face is considered a high-level modebecause it resonates to any face (but tonothing else), and does so regardless of theidentity, expression, orientation, or illumi-nation of that face, whereas a low-level moderesonates to detailed local features of lightness,color, texture, orientation, and so on, whichare possessed by only a few faces in a fewposes, and perhaps by some stimuli that arenot faces at all.

As is indicated by phenomena of perceptualcompletion, excitation of a mode tends toinduce sympathetic activity in other modes.When these other modes are "above" or"below" the initially excited mode, we havewhat information-processing theorists referto as bottom-up and top-down processes.However, in accordance with Gibson's radicalinsight, a high-level mode may resonate toan abstract external invariant directly; itsexcitation need not depend on excitation ofmodes that are lower in the hierarchy andthat correspond to more elementary featuresof the external object or event (cf. Runeson,1977; and the further discussion in Pomerantz& Kubovy, 1981). Neisser (1976, pp. 112-113) characterized the essential relation be-tween different levels of such a hierarchy asone of nesting or embedding rather than oneof causation.8

8 More accurate than my implied one-dimensionalhierarchical scheme, ranging from abstract and conceptualto concrete and sensory, would be a two-dimensionaltriangular scheme in which the three corners represent(a) abstract concepts (e.g., face, smile, triangle, or rotation),(b) concrete percepts (e.g., John's smiling face or a blue


Externally and Internally InstigatedRepresentational Processes

In Figure 7, I use a vertical rectangle torepresent the hierarchy of resonant modes,ranging between those that are most abstractand conceptual, at the top, and those that aremost concrete and sensory, at the bottom.Each triangle represents a currently excitedmode of the system. I assume that the systempreserves no record of the sources of excita-tion of any mode, which could be primarilyfrom within the system (whether from aboveor below) or from without. To show how thesame system may be differently excited inexperiencing sensations and in perceiving,dreaming, hallucinating, imagining, or think-ing, I have nevertheless distinguished theactive modes in Figure 7 according to whetherthe primary sources of their excitation wereexternal (triangles pointing up) or internal(triangles pointing down).

Because unstructured stimuli (includingdirect mechanical, electrical, or chemical ir-ritations of sensory pathways or their corticalprojection areas) are not matched to higherlevel resonances, they produce only themeaningless "lights, colors, forms, buzzes,hums, hisses, and tingles" (see Penfield, 1958,pp. 11-13) that correspond to low-level res-onances of the system (as illustrated in Figure7, Rectangle a). In contrast, perception ofmeaningful external objects and events ariseswhen resonant activity is induced at all levelsof the system (as in Figure 7, Rectangle b).

Even when there is no external input,resonant modes may still become sponta-neously excited. Subjective reports, supportedby some neurophysiological evidence (e.g.,Dement, 1965; Penfield, 1958; West, 1962),suggest that when the system becomes func-tionally decoupled from sensory input duringREM sleep or perhaps in hypnagogic, hyp-nopompic, or hallucinatory states, even thelowest level resonances may become entrainedby higher level activity (as depicted in Figure

equilateral triangle rotating clockwise), and (c) sensations(e.g., flashes, colors, buzzes, or tingles). For simplicity ofexposition here (and in Figure 7), I have in effectcollapsed such a triangle into a one-dimensional (rect-angular) scheme by compressing the "percept" cornertoward the opposite side, halfway between concepts andsensations.

PERCEIVING DREAMING IMAGINING THINKINGor (in

REMEM- nonverbal,BERING "imagelesa"

thoughts)

IONS (under (under re- HALLUfavorable duced or NATINconditions) ambiguous

conditions)

AA

A

4*

ÂAA A*A

A^

*T»

T*

TAA TTTAAA

AAA

\;

T^V

T T

T^


thought, and dreaming and hallucination.According to Freud, in dreams and halluci-nations the normal direction of flow fromthe perceptual system through memorial andideational systems (and ultimately to motoractions) is reversed: There is "a retrogressionin the psychic apparatus from some complexact of ideation to the raw material of thememory-traces which underlie it" (p. 398)until we have, in the end, "thoughts trans-formed into images" (p. 400). But in "inten-tional recollection and other component pro-cesses of our normal thinking . . . duringthe waking state this turning backwards doesnot reach beyond memory-images; it is in-capable of producing the hallucinatory revivalof perceptual images" (Freud, 1931, p. 398).

Perceiving under reduced or ambiguousconditions (as at night, in the psychologicallaboratory, or in looking at pictures) is inter-mediate between normal perceiving and pureimagining. Although many of the modes aredirectly excited by the externally availableinformation, other modes, perhaps especiallyat higher levels, are sympathetically excitedby purely internal activities corresponding toset, expectation, and bodily state. The finalresult tends to be the overall pattern ofmutually resonant activity that is most inter-nally consistent throughout all levels of thesystem.

Thus, although J. Gibson (1970) held thatperceiving is an entirely different kind ofactivity from thinking, imagining, dreaming,or hallucinating, I like to caricature percep-tion as externally guided hallucination, anddreaming and hallucination as internally sim-ulated perception. Imagery and some formsof thinking could also be described as inter-nally simulated perceptions, but at more ab-stract levels of simulation.

Resonance as a Spontaneous Emergentin Neural Networks

In proposing resonance as a metaphor, Ihave intentionally remained noncommittalconcerning neural mechanisms. In particular,I am not claiming that resonant activity inthe brain is necessarily a periodic oscillationlike the vibration of a string, although thereis evidence that some responses of the per-ceptual system do have this character (e.g.,

Bialek, 1983; Freeman, 1975). Certainly, themost concrete interpretation of the metaphor,in terms of conservation and dissipation ofenergy, would be inappropriate. Independentlyof sensory input, the brain's metabolism con-tributes energy to the process of perceptualinterpretation and, hence, provides for a kindof amplification not found in a passive reso-nator. The resonance I speak of is thereforenot strictly a resonance to energy; as J.Gibson (1966) implied, it is a resonance toinformation. Subject to this proviso, however,resonance may arise in the brain in a moreliteral sense than I have so far claimed.

Such a notion of neural resonance has avariety of precursors going back over 30 yearsto related ideas of cortical standing waves orinterference patterns (e.g., Lashley, 1942), re-verberatory circuits (e.g., Ashby, 1954; Rash-evsky, 1948), and especially, reverberatorycell assemblies and phase sequences (Hebb,1949). Moreover, the concept of resonanceitself has come into recent prominence inconnection with biology and communication(Thorn, 1972/1975, pp. 134, 145), brain sci-ence (Changeux, 1983), and sensory psycho-physiology (Ratliff, 1983). In the latter con-nection, vision researchers have of courselong referred to the tuning curves of individualreceptor mechanisms. Indeed, in receivingthe 1983 Pisart Vision Award, Floyd Ratliffwent as far as to say that "the neural networksin our visual systems are not only figurativelytuned, they are literally harmonic systems"(Ratliff, 1983, p. 10).10

Moreover, mathematical analyses have in-dicated that resonance is to be expected as anatural emergent in neural networks. Drawingon the theory of linear systems, Greene(1962a, 1962b) showed that if information isrepresented by graded signals (e.g., by den-dritic potentials and by rates of axonal firing),

10 In connection with the possible harmonic nature ofthe visual system, I find it amusing that many visionresearchers, who (unlike Ratliff) insist on an exclusivelynonmetaphorical approach to their subject, neverthelessuse the term octave to refer to a two-to-one ratio ofspatial frequencies. Perhaps they have not noticed thatthe term does not derive from a physical relation betweenfrequencies but from the fact that the seven-tone diatonicmusical scale (which is determined more by cognitivethan by physical constraints; see Balzano, 1980; Shepard,1982a) returns to the tonic with the eighth step.


then even networks composed only of linearelements will possess characteristic reso-nances, unless "special apparatus is installedto suppress the transients and resonantmodes" (Greene, 1962a, p. 257). He con-cluded that "evolution either suppressed thisfeature or exploited it. Since its propertiesresemble those of animal behavior, the lattermight be suspected" (Greene, 1962b, p. 395).

Some of the properties that Greene (1962b)derived for nonhomogeneous neural networksare (a) that "resonant modes . . . selected(through natural selection) as the bearers ofmeaningful information . . . will tend to bestabilized against random disturbances" (p.409) and (b) that "extremely complex config-urations may be represented by a small num-ber of simple intensities," enabling the organ-ism to "switch from one highly integratedbehavior pattern to another, without seemingto be required to make adjustments in themultitude of parameters that would be nec-essary to specify all the individual parts ofthe patterns" (p. 407). Essentially the sameproperties have more recently been deducedfor hierarchical, nonlinear neural networksby Grossberg (1980), who similarly concludedthat "adaptive resonances are the functionalunits of cognitive-coding" (p. 29).

Greene (1962b), Grossberg (1980), andAnderson, Silverstein, Ritz, and Jones (1977)cited categorical perception and the perceptualreversals experienced while viewing ambigu-ous stimuli as examples of the global switchesfrom one meaningful organization to anotherthat are expected to arise in a resonantnetwork. Of complex resonant systems ingeneral, Rene Thorn said "Usually the systemmust choose from several possible resonances,and this competition of resonances has neverbeen studied mathematically, even though itseems to be of the greatest importance"(Thorn, 1972/1975, p. 134). Nevertheless,Greene (1962a), within the context of theparticular linear systems investigated by him,derived the reorganizations that occur withincreasing speed in the gait of a quadrupedas but another manifestation of discontinuousshifts between resonant modes. Shifts of thislast type seem suggestively like the discontin-uous changes in perceptual organization (e.g.,from whirling jump rope to spinning disk)that Zare and I found to occur as the alter-

nation between the apparent motion displaysof Figure 6 (Panels a and b) is graduallyaccelerated.

Apparently then, resonant modes mightnaturally arise in neural circuits and mightcontribute to coherence both among internalprocesses and between those processes andsignificant environmental constraints. Thenotion of resonance that I have been advo-cating differs from the earlier proposals con-cerning cortical waves and reverberatory cir-cuits in focusing less on the form of theneural activity itself and more on the func-tional mesh (Shepard, 1981b) that must holdbetween that activity and the external objectsand events that it represents. Admittedly, wehave yet to determine exactly how kinematicgeometry is embodied in the resonant modes.Nevertheless, if the characteristics of thesemodes are determined by the connectivityand synaptic transmission coefficients of theneural circuit, there is no obvious reasonwhy evolutionary selection and individuallearning could not shape a circuit with therequisite properties."

The most radical departure from themechanisms usually proposed for perceptualprocessing is motivated, here, by the desireto encompass dreaming, imagining, andthinking. Even the undamped strings of apiano, in the absence of an acoustic stimulus,begin to vibrate only when struck by theircorresponding padded hammers—events thatdepend on the intervention of an agencyexternal to the piano itself. Has the homun-culus of the picture-in-the-head metaphor ofperception and imagery (although differentlyguised as performer rather than perceiver)reared again its ugly head? Not necessarily.The strings of an aeolian harp, for example,are excited not by intentional hammer blowsbut by random gusts of wind. Yet the ensuingactivity is harmonious—provided, of course,that the strings have already been properlytuned.

" See, for example, Changeux (1983). Note that theisomorphism required between the external constraintsand their internal representation need be no more thanan abstract or second-order type (see Shepard, 1975) andthat related, spatially distributed parallel processes (suchas relaxation methods) for satisfying constraints havealready been shown to be effective for scene analysis incomputer vision (e.g., see Rosenfeld, 1982; Waltz, 1975).


Similarly, the natural modes of resonanceof a neural network, if they are already tunedto naturally occurring external objects andevents, tend to ring and hence to representsuch objects and events when, in the REMstate, under the influence of fever or drugs,or, to some extent, during mere idle medita-tion, the network is played on by randominternal "gusts" of neural or chemical exci-tation. Of course, to the extent that eitherabstractly formulated goals or pressing bodilystates become ascendent, they tend to entrainthe entire ongoing activity into a more di-rected course.

In his acceptance address, Ratliff (1983)noted that a resonance theory of thinkingcan in fact be traced back at least to Coleridge,who, in his 1795 poem "The Eolian Harp,"after writing (lines 13-17)

"that simplest Lute,Placed length-ways in the clasping casement,

hark!How by the desultory breeze carress'd . . .It pours forth such sweet upbraiding,"

went on to ask

"And what if all of animated natureBe but organic Harps diversely fram'd,That tremble into thought?"

(lines 44-46; see Richards, 1950, pp. 65-67).

Implications Concerning Robustnessof Brain Function

I have suggested that what is being repre-sented within the system co

Shepard (1984) Ecological constraints on internal ...wexler.free.fr/library/files/shepard (1984)...

Documents

Transcript of Shepard (1984) Ecological constraints on internal ...wexler.free.fr/library/files/shepard (1984)...