Internal models, adaptation, and uncertainty

Reza ShadmehrJohns Hopkins School of Medicine

Ali Ghazizadeh

Maurice Smith

Konrad Koerding

Siavash VaziriJoern Diedrichsen

Duhamel, Colby, & Goldberg Science 255, 90-92 (1992)

Internal models predict the sensory consequences of motor commands

musclesMotor commandsforce

Body partState change

Sensory system

ProprioceptionVision

Audition

Measured sensory

consequences

Forward model

Predicted sensory consequences

Integration

Bayesian

mixture

Vaziri, Diedrichsen, Shadmehr, J Neurosci 2006

Reach endpoints with respect to targetTime (msec)

Vaziri, Diedrichsen, Shadmehr, J Neurosci 2006

Variance in reach errors indicates an integration of the predicted and actual sensory consequence of oculomotor

commands

Motor commands

Sensory system

Measured sensory input

Forward model

Predicted sensory consequences

Integration

Estimate of target location

Motor commands musclesforce

Body partState change

Sensory system

ProprioceptionVision

Audition

Measured sensory

consequences

Forward model

Predicted sensory consequences

Integration

Bayesian

mixture

What are internal models good for?

Improve ability to sense the world. By predicting the sensory consequences of motor commands, and then integrating it with the actual sensory feedback, the brain arrives at an estimate that is better than is possible from sensation alone.

Equivalent to muscles being too strong

McLaughlin 1967

TargetEye X

30%

Saccadic target jump experiments: gain reduction

Kojima et al. (2004) J Neurosci 24:7531.

Result 1: After changes in gain, monkeys exhibit recall despite behavioral evidence for washout.

+ _ + _ + _

Savings: when adaptation is followed by de-adaptation, motor system still exhibits recall

Saccade gain = Target displacement

Eye displacement

Result 2: Following changes in gain and a period of darkness, monkeys exhibit a “jump” in memory.

+ _ +

Offline learning: with passage of time and without explicit training, the motor system still appears to learn

Kojima et al. (2004) J Neurosci 24:7531.

( ) ( )( )1 2

( ) ( ) ( )

( 1) ( ) ( )1 11 1

( 1) ( ) ( )2 22 2

ˆ ˆ ˆ

ˆ

ˆ ˆ

ˆ ˆ

n nn

n n n

n n n

n n n

y y y

y y y

y a y b y

y a y b y

Motor adaptation as concurrent learning in two systems:A fast learning system that forgets quicklyA slow learning system that hardly forgets

Smith, Ghazizadeh, Shadmehr PLOS Biology, 2006

prediction

Prediction error

Learning

Savings: de-adaptation may not erase adaptation

Task reversal periodre-adaptation

Trial number

Smith, Ghazizadeh, Shadmehr PLOS Biology, 2006

A

1w

y

x

Hid

den

sta

tes

Context

perturbation

2w

mw

t

1w

y

x

2w

mw

t

Slow change

fast change

The Bayesian learner’s interpretation of prediction error

Offline learning: Passage of time has asymmetric affects on the fast

and slow systems

Smith, Ghazizadeh, Shadmehr PLOS Biology, 2006

Task reversal period

“dark” period

re-adaptation

Trial number

Slow stateFast state

-

( 1) ( ) ( )1 11 1

( 1) ( ) ( )2 22 2

ˆ ˆ

ˆ ˆ

n n n

n n n

y a y b y

y a y b y

1. Perturbations that can affect the motor plant have multiple time scales.Some perturbations are fast: muscles recover from fatigue quickly.Some perturbations are slow: recovery from disease may be slow.

2. Faster perturbations are more variable (have more noise).

3. Disturbances result in error, which can be observed, but with sensory noise.

4. The problem of learning is one of credit assignment: when I observe a disturbance, what is the time-scale of this perturbation?

5. To solve this problem, the brain must keep a measure of uncertainty about each possible timescale of perturbation.

The learner’s view about the cause of motor errors

( ) (1 1/ ) ( )disturbance t disturbance t

0, /N c

( ) ( )observation t disturbance t

Koerding, Tenenbaum, Shadmehr, unpublished

Savings: de-adaptation does not washout the

adapted system

Simulation

Koerding, Tenenbaum, Shadmehr, unpublished

Spontaneous recovery

Model 1 (Smith et al.): Error causes changes in multiple adaptive processes. Fast adaptive processes are highly responsive to error, but quickly forget. Slowly adaptive processes respond poorly to error, but retain their changes.

Prediction: When actions are performed with zero error, states of the adaptive processes decay, but at different rates.

Model 2 (Koerding et al.): Motor system is disturbed by processes that have various timescale (fatigue vs. disease). Credit assignment of error depends on uncertainty regarding what is the timescale of the disturbance.

Prediction: When there are actions but the sensory consequences cannot be observed, states decay at various rates, but uncertainty grows. Increased uncertainty encourages learning.

What prediction dissociates the two models?

Trial number

Slow stateFast state

Task reversal period

“dark” period

re-adaptation

-

Model 1: After a period of “darkness”, there will be spontaneous recovery, but rate of re-adaptation will be the same as initial learning.

Adapting without uncertainty

Smith, Ghazizadeh, Shadmehr PLOS Biology 2006

Model 2: After a period of “darkness”, there will be spontaneous recovery, but the rate of re-adaptation will be faster than initial learning.

Adapting with uncertainty

Monkey data from Kojima et al. (2004). Simulations from Koerding, Tenenbaum, Shadmehr, unpublished

Bayesian learner

1 1000 2000 3000

Saccade number

Dark

ness

Robinson et al. J Neurophysiol, in press

Sensory deprivation may increase uncertainty, resulting in faster learning

Monkeys were trained each day, but between training sessions they put on dark goggles, reducing their ability to sense consequences of their own motor commands.

Dark

ness

Adapting with uncertainty: some predictions

Sensory deprivation Faster subsequent rate of learning.

Example: A subject that spends a bit of time in the dark will subsequently learn faster than a subject that spends that time with the lights on.

Why: In the dark, uncertainty about state of the motor system increases.

Longer inter-stimulus interval Better retention.

Example: A subject that trains on n trials with long ITI will show less forgetting than one that trains on the same n trials with short ITI.

Why: events that take place spaced in time will be interpreted as having a long timescale.

Ali Ghazizadeh Maurice Smith

Konrad Koerding

Siavash VaziriJoern Diedrichsen

By combining the predictions of internal models with sensory measurements, the brain ends up with less noisy estimates of the environment than is possible with either source of information alone.

Fast and slow adaptive processes arose because disturbances to the motor system have various timescales (fatigue vs. disease). When faced with error, the brain faces a credit assignment problem: what is the timescale of the disturbance? To solve this problem, the brain likely keeps a measure of uncertainty about the timescales.

A prediction error causes changes in multiple adaptive systems. Some are highly responsive to error, but rapidly forget. Others are poorly responsive to error but have high retention. This explains savings and spontaneous recovery.

Summary