Computational Maps in the Visual Cortex

Risto Miikkulainen

Department of Computer Sciences

and Institute for Neuroscience

The University of Texas at Austin

Joint work with Jim Bednar, Yoonsuck Choe, and Joseph Sirosh

Supported in part by NIMH 1R01-MH66991, NSF IIS-9811478 & IRI-9504317

Understanding Vision

(Logothetis 2002)

• How is a system as complex as the human visual system constructed?

• How can it be both genetically and environmentally determined?

• How does its structure support functions such as perceptual grouping?

Role of Computational Modeling

V1

LGN

ON OFF

Retina

• Computational model is an artificial subject with full access– Test hypotheses computationally, make predictions

• Computational theory of the visual cortex– Build better artificial systems– Improve medical treatment

Human Visual System

chiasmOptic

Right

Left

Left LGN

Right LGN

Visual field

Left eye

Right eye

(V1)cortexvisualPrimary

• Retina, LGN, V1...etc.

• Structure well known

Receptive Fields

Retina

ON/OFF

OFF/ON

LGN

ON/OFF

OFF/ON

V1

Simple 2-lobe

Simple 3-lobe

Spatiotemporal

• Center-surround; static and moving lines; combinations

Columnar Organization of V1

(Kandel et al. 1991)

• Roughly hierarchical ordering:– Retinotopy, OD, OR, DR– Color, spatial frequency, disparity?

• Within column, similar responses: 2D structure

Measuring Cortical Maps

• Surface reflectance changes with activity

• Optical imaging can be used to detect

Orientation Map

(7.5 mm × 5.5 mm in macaque V1; Blasdel, 1992)

• Preferences mapped systematically

• Linear zones, pinwheels, saddles, fractures

Orientation & Ocular Dominance Map

(4 mm × 3 mm in macaque V1; Blasdel, 1992)

• Systematic interactions

– OD, OR boundaries at right angles

– Pinwheels, saddles in the middle

Orientation & Direction Map

(1.4 mm × 1.1 mm in ferret V1; Weliky et al. 1996)

• Systematic interactions

– OD, OR boundaries at right angles

– OR patches contain opposite DR

Lateral Connections

(2.5 mm × 2 mm in tree shrew V1; Bosking et al. 1997)

• Link to similar responses

• Patchy structure, extend along OR preference

Development

(4 mm × 3 mm OR+select in ferret V1; Chapman et al. 1996)

• Structure emerges during development

• Some prenatally, much postnatally

• How and why?

LISSOM Model

Right retinaLeft retina

3

2

1

0

OFFONOFFON

LGN

V1

• Combined OR, OD, DR

• Retina, LGN, V1 (+ other areas)

• 2D sheets, afferent and lateral connections

• Hebbian learning in V1

Activation

Retinal activation LGN response V1 initial V1 settled

• Luminance adjustment in retina

• Sharpening in LGN (ON−OFF shown)

• Settling in V1:

η′i = σ (ΣkχkAki +ΣjηjEji − ΣjηjIji)

Adaptation

Itera

tion

0Ite

ratio

n10

,000

Afferent Lateral excitatory Lateral inhibitory

• Normalized Hebbian learning: A′ki =

Aki+αχkηi

Σmn(Aki+αχkηi)

→ Input-driven self-organization

• Pruning unused connections

• Results in realistic receptive fields, patchy lateral connections

Orientation Map

OR+selectivity, iteration 0 OR+selectivity, iteration 10,000

• Systematic preferences emerge

• Similar structures as in biology

Orientation & Ocular Dominance Orientation & Direction

OR & OD & DR Map Lateral Connections

Wei

ghts

His

tC

onne

ctiv

ity

Iso-OR patch Pinwheel Saddle Fracture

• Link similar responses• OR primary factor• Matches biology; detailed predictions

Self-Organization Conclusions

• How is V1 constructed?

– Input-driven self-organization

• Predictions:

– Input deprivation (e.g. strabismus)

– Connection patterns

– Plasticity

– Illusions and aftereffects

– Visual coding

What Is the Goal of Visual Coding?

V1

LGN

ON OFF

Retina

• Representing the important features of the input

• Efficient use of resources:

Can represent more information within a limited system

How is Such a Coding Constructed?

Initial response Redundancy-reduced

sparse response

• Not by reducing units: V1 is much larger than the retina

• Could be a sparse code with few active units

• Need to make sparse by reducing redundancy(Barlow 1972; Atick 1992; Field 1994; Simoncelli & Olshausen 2001)

Lateral Connection Hypothesis

(2.5 mm × 2 mm in tree shrew V1; Bosking et al. 1997)

• Afferent connections respond to input features

• Inhibitory lateral connections decorrelate the response

– Connect neurons that respond to similar inputs

– Response of one neuron can be predicted from the other

– Can be suppressed without losing information

Testing the Hypothesis

V1

LGN

ON OFF

Retina

• Difficult to test experimentally

– Requires many neurons, short time scales

• Can be tested in computational models

Does LISSOM Form a Sparse Code?

Retinal activation Initial V1:

Kurtosis 37.9

Settled V1:

Kurtosis 63.0

• Self-organize a LISSOM map

• Measure kurtosis of the response

• → The settled response is sparser

Does LISSOM Reduce Redundancy?

Retinal activation Reconstruction

from initial V1:

Avg. RMS error 0.094

Reconstruction

from settled V1:

Avg. RMS error 0.094

• Reconstruct the input from V1 activity

• Nonlinear: train a backprop net to map back

• → No information lost

Is Self-Organization Necessary?

Retinal activation Settled V1:

Kurtosis 63.0

SoG-settled V1 :

Kurtosis 60.1

• Isotropic (Sum-of-Gaussians; SoG) lateral

connections instead

• Can be adjusted to match kurtosis

• → Sparse code can be formed

Is Self-Organization Necessary?

Retinal activation Reconstruction

from LISSOM V1:

Avg. RMS error 0.094

Reconstruction

from SoG V1:

Avg. RMS error 0.137

• Reconstruction no longer works!

• Information reduced, not just redundancy

• → Self-organization is necessary

• → Forms a sparse, redundancy-reduced code

Nature vs. Nurture

(Johnson and Morton 1991)

• Development through input-driven self-organization

• But some order appears innate

– E.g. orientation maps

– E.g. newborn face preferences

Newborn Face Preferences

0o

10o

20o

30o

40o

50o

60o

Fin

al tr

acki

ng o

rient

atio

nF

Within 1 hour

0o

10o

20o

30o

40o

50o

60o

Fin

al tr

acki

ng o

rient

atio

nF

21 hours(Johnson et al. 1991)

• Significant preference for face-like schematics

• Genome too small to specify connectivity, behavior

• Three-dot patterns strongest; why?

Retinal Waves

(1 mm × 1 mm in ferret retina; Feller et al. 1996)

• Traveling waves in the retina before birth

• Could serve as input for self-organization

PGO Waves

V1

LGN

(Marks et al. 1995)

• Ponto-geniculo-occipital waves

• Shape unknown, but activates V1

• Could introduce the three-dot bias

HLISSOM Model

LGN

ON OFF

V1

Face−selective area (FSA)

Retina PGOpattern

generator

• Include PGO & FSA

sheets

• Three-dot input patterns

in PGO

• Study prenatal and

postnatal

self-organization

Newborn LISSOM Face Preferences

Ret

ina

LGN

V1

FS

A

2728

0.0

927

5.8

836

5.5

664

4.0

1094

0.0

994

0.0

931

0.0

807

0.0

275

0.0

• Matches newborn preferences in every known case

Newborn LISSOM Face Preferences (2)

Ret

ina

LGN

V1

FS

A

4756

11.7

5778

7.5

6932

6.1

4797

11.1

4796

0.0

7614

0.0

7858

0.0

5072

0.0

4845

18.1

• Prefers top-lit faces; not objects

• Images not tested on infants

Effect of Pattern Types

PG

OLG

NF

SA

-RF

Sch

Img

1.0

1.0

0.8

1.0

0.8

0.8

0.3

0.9

0.5

0.4

• Three dots not the only possible pattern

• Not all patterns work

Pattern Generation Conclusions

• How are nature and nurture combined?

– Through internal pattern generation

• Predictions

– Types of internal patterns

– Postnatal decline of preferences

– Holistic perception of the face develops

– Mother preferences develop

Perceptual Grouping

Proximity Good continuation World knowledge

• Perceiving whole objects

• Low-level based on “Gestalt” principles

• Mediated by lateral connections in V1?

PGLISSOM Model

ON

V1

Retina

GMAP

SMAP

• Self-organization needs long-range

inhibition

• Grouping needs long-range

excitation

• → 2-layer model of the column

Leaky Integrator Neuron

χ~

η~

η~

ζ~

Inhibitory lateral connections

Excitatory lateral connections γ

γ

γ

γ

A

E

I

C

v S

A

σ

a

r

E

I

b

γ

C

θ

Σ

Σ Σ

Spike generator

Intra−columnar connections

Afferent connections

Σ

Σ

Σ

• Binding and segmentation by synchronization

• Need spiking neurons

Self-Organized Lateral Connections

100

10

1

0.1

0.01

δ

Likelihood ratio

= 1.23o

φ = 90o

φ = 0o

0.00010.001

0.010.1

1

δ

Relative probability

= 0 o

= 90 o

φ

= 27

φ

Edge cooccurrence in nature(Geisler et al. 2001)

Lateral excitation in GMAP

• PGLISSOM self-organizes like LISSOM

• Lateral connections match visual environment

Contour Integration Process

1

2

36

54

7

8

9

1

2

3�

4

5�

6

7

8�

9�

0�

100 200�

300�

400�

500�

0◦ of orientation jitter

21

3

5 6

98

7

4 1

2

3�

4

5�

6

7

8�

9�

0�

100 200�

300�

400�

500�

50◦ of orientation jitter

• Synchronizes continuous contours

• Depends on how “good” the contour is

PGLISSOM vs. Human Performance

0o

10o

20o

30o

40o

50o

60o

70o

60%

70%

80%

90%

100%

Hum

an: A

ccur

acy

(por

tion

corr

ect)

Subject WSG

0o

10o

20o

30o

40o

50o

60o

70o

Orientation jitter

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Mod

el: c

orre

latio

n (r

)

Model

• Depends on jitter like human performance

Contour Segmentation

12

3

4

5

6

7

89

1

2

3�

4

5�

6

7

8�

9�

0�

100 200�

300�

400�

500�

• Multiple contours by alternating

• Upto 5-9 contours

Contour Completion

123

45

1

2�

3�

4�

5�

0�

100 200 300�

400 500�

• Filling in gaps

• Basis for edge-induced illusory contours?

Illusory Contours

Open1

23 4

56789

10

1112 15

13

14

123

�45

�678

�9

�101112131415

0�

100 200 300�

400 500�

Closed1

23 4

56789

10

1112 15

13

14

123

�45

�678

�9

�101112131415

0�

100 200 300�

400 500�

• Kanizsa: proximity & continuation

• Closed contours easier

• Matches human performance

Perceptual Grouping Conclusions

• How does the structure support functions like grouping?

– Synchronization mediated by self-organized lateral connections

• Predictions:

• Effect of activation decay, noise, refractory period on synchronization

• Image statistics → lateral connectivity → performance

– Frequency, curvature, etc. differ across visual fields

– Performance differs in fovea vs. periphery, upper vs. lower hemifield

Future Work

• Self-organization

– Color, frequency, disparity

– Hierarchy, feedback, multimodal integration

• Development

– Characterizing internal patterns

– Constructing complex systems

• Grouping

– Verify synchronization hypothesis with TMS

– Line-end-induced illusions in V2?

Topographica

• General simulator for cortical maps (v0.8.2 Feb 2006)

Conclusion

• Wealth of data + powerful computing available

• Neuroscience research in vitro, in vivo, in silico

• Computational theory of the visual cortex

– Continuously adapting self-organizing system

– Shaped by internal and external input

– Lateral connections play a major role

• Exciting possibilities for future work

Further Details

(Springer, 2005)

Demos, software, etc.:www.computationalmaps.org