Classical conditioning via inference over observable situation contextsNisheeth Srivastava (nsrivastava@ucsd.edu)

Department of Psychology, University of California San DiegoSan Diego, CA 92093 USA

Paul R Schrater (schrater@umn.edu)Department of Psychology, University of Minnesota

Minneapolis, MN 55455 USA

AbstractIn this paper, we demonstrate that predicting stimulus co-occurrence patterns in a Bayes-optimal manner endogenouslyexplains classical conditioning. Simulated experiments witha standard Bayesian implementation of this model show thatit is capable of explaining a broader range of effects than anyprevious theory of classical conditioning. By simplifying themathematical structure of statistical modelling of conditioningand demonstrating its ability to explain a large set of experi-mentally observed effects, our work advances Bayes-optimalinference about stimulus co-occurrence as a rational principleexplaining classical conditioning.Keywords: learning; Bayesian modelling; computer simula-tion; decision-making

IntroductionClassical conditioning is a form of associative learningwherein animals are trained to respond positively or nega-tively to some stimulus towards which they were previouslyneutrally disposed. The classic example of such condition-ing arises from its accidental discovery by Ivan Pavlov, whodiscovered that dogs kept in his lab to be used as experi-ment subjects would salivate as if they were about to feed atthe sound of a bell that typically heralded mealtime (Pavlov,1927). Such transference of somatic response from a posi-tive stimulus (food) to a prima facie neutral one (bell sound)through situational co-occurence has since been called classi-cal (or Pavlovian) conditioning. In any classical conditioningexperiment, a stimulus for which an animal is known to havea positive or negative response (called unconditioned stim-ulus, or US) is paired with a neutral one (called the condi-tioned stimulus, or CS) until the animal transfers the somaticresponse typically evinced when encountering the US to en-counters with CS. When such a conditioned response (CR) iselicited, the animal is considered to have been conditioned.

Despite the importance and long lineage of research inclassical conditioning, formal theories still find it difficult toexplain a wide variety of conditioning-related behaviors seenin experiments. The Rescorla-Wagner model (Rescorla &Wagner, 1972) is the most influential formal model of con-ditioning, but cannot discriminate non-linear cue combina-tions and other interesting conditioning effects. The com-parator class of theories has multiple formal instantiations,all of which share in the intellectual attempt to coopt en-vironmental cues beyond the labeled CS to help predict theUS. Attention-based models explain conditioning effects us-ing information-theoretic measures of informativeness of var-ious CS. Both these latter theories can explain some things

that Rescorla-Wagner can’t, but are perceived to have theirown limitations, particularly in that they remain formally un-derspecified. Thus, there is no theoretical consensus on mod-elling classical conditioning, and until recently, little hope ofarriving at one.

Recently, sophisticated modelling by Courville etal (Courville, Daw, & Touretzky, 2006) and (Gershman &Niv, 2012) has shown that comparator-type theories formallyinstantiated as Bayesian inference about the latent structureof an animal’s environment have substantial predictivepower. These formal models statistically ground previouslysomewhat heuristic presentations of the comparator hy-pothesis (Stout & Miller, 2007) by using animals’ knownpropensity to make situational frequency judgments (Hasher& Zacks, 1984) to inform conditional probability estimatesof the form p(US|CS,US,S) , where S are non-CS en-vironmental factors that can help predict US. Simulationstudies demonstrate that such Bayesian latent variablecomparator models explain substantially more conditioningeffects than earlier less formal comparator accounts. Forinstance, Courville et al. (2006) review a large number ofstudies documenting effects that have traditionally beenexplained using attentional models, and demonstrate thateach one of them can be explained using a latent variablemodel that postulates that animals try to infer what causecould have led to their current stimulus set observation.Gershman and Niv (2012) present a different latent causemodel and demonstrate its utility in explaining a number ofconditioning effects, including ones that comparator theorieshave erstwhile been hard pressed to explain. These mod-elling advances have made a concrete case for investigatingcontext-sensitive US prediction as a primary explanation forclassical conditioning.

However, while latent cause modelling is a promising newdirection, the theory is still immature, with a number of dif-ferent formal specifications possible, each with a different setof potentially implausible statistical assumptions. For exam-ple, (Courville et al., 2006) assume that animals learn therate of change of parameters for a fixed set of latent mod-els. In contrast, (Gershman & Niv, 2012) use infinite mixturemodelling to infer latent causes dynamically, but are forcedto treat causes as clusters of observation vectors, assume aDirichlet process prior on the set of causes, and impute onlyMAP estimation to the cause inference. It is not currentlyclear which assumptions in these models are essential to the

1503

theory, which are simply modelling requirements, and whichare entirely superfluous.

In this paper, we adopt a simpler third interpretation forthese latent variables, viz. that they are not latent but rathersimply index stimuli co-occurrences the animal has observedin the past. We show that this interpretation leads to a simpleformal specification of the comparator account of classicalconditioning that expands upon the explanatory power of la-tent cause-based models, while additionally being observableand directly testable. As evidence for the predictive abilityof our model, we present simulated replications of ten classicconditioning effects by agents using our model. Its successfurther strengthens the case for normative Bayesian inferenceas a rational basis for animal behavior (Knill & Pouget, 2004).

The situation prediction modelThe fundamental novelty of our approach to modelling clas-sical conditioning is that we don’t try to model it. Instead, wetry to model what we believe is a superordinate goal for ani-mals - predicting situations and their corresponding task-sets.We consider it more ecologically natural to think of animalsbeing driven by the need to orient themselves to their envi-ronments by constantly making adaptive predictions. If weassume that animals do partition the environment as discretesituations, trying to predict which one they are in becomes anatural and essential goal. It is this essential process that wetry to model.

We model this basic orienting process as sequentialBayesian updating, where animals use evidence about stimu-lus co-occurrences observed in the past o1:t−1 to predict thelikelihood of being in situation context c and having to re-spond to it at time t. The standard Bayesian formulation ofthis inference is,

p(c|o1:t) ∝ p(ot |c)× p(c|o1:t−1). (1)

To specialize this model to the classical conditioningparadigm, we simply define observations in a specific way.We assume that, in addition to generic environmental stim-uli S, the animal can observe CS and US in the condition-ing setup. We therefore model an observation at time t asot = {US,CS,S}(t), where each stimulus in turn is a binaryattribute which is either present or not present in the obser-vation vector ot . We interpret c as the observation-indexedsituation context within which an animal has to respond tostimuli that are biologically relevant (US). Formally, we sim-ply index c using o, thereby instantiating C as the set of allprevious observations unique with respect to observation setmembership.

With these additional assumptions constraining our inter-pretation of the model defined in (1), we obtain a specializa-tion calculating p(c|{US,CS,S}(1 : t)) as,

p(c|{US,CS,S}(1 : t)) ∝ p({US,CS,S}(t)|c) (2)× p(c|{US,CS,S}(1 : t−1)).

The key quantity of interest is the likelihood termp(US(t),CS(t),S(t)|c). Both within the temporal constraintsof conditioning experiments and in the real world, animalswill typically encounter situations where biologically irrel-evant stimuli S and CS are observed, and biologically im-portant US is to be predicted, making the separate computa-tion of p(US|CS,S) ecologically natural. It is easy to derivethe probabilistic strength of association p(US|CS) from thiscomputation, and we believe that the stimulus specific CRsmeasured in conditioning experiments are simply behavioralmeasurements of this quantity. Thus, we see that our likeli-hood term matches the interpretation of CR in the condition-ing paradigm, resulting in an overall interpretation of condi-tioning responses as playing a facilitative role in the animal’sdeeper goal of situational context prediction.

In our representation, each context has an associated set ofstimuli c j ∈ c, and the probability that an incoming observa-tion vector o is generated from this context simply requiresan assessment of set membership mismatches. Let p(oi|c j)be the probability that the ith element of the observation vec-tor o matches the jth element of c. Then, the probability thatstimulus x = oi is generated by an object set c is,

p(x|c) = 1− β

|c|

|c|

∑j(1−δ(x− c j)) (3)

where β is a parameter controlling the magnitude of thepenalty imposed for each mismatch observed, with p(x|o)similarly defined.

Predictive evidence for US associated with each contextp(US|x,c) is imputed individually. As a rule of thumb, if acontext does not contain US, it is unlikely to predict futureUS and vice versa. This evidence is compiled across all pre-viously inferred c ∈ C to predict strength of association withUS for any x, conditional on it being present in the currentobservation vector ot . While only the CS is relevant for mod-elling conditioning, our model is designed to predict strengthof association with all possible stimuli. In settings where theenvironmental stimulus set S is fixed, our model predicts thatthe conditioning response CR will be proportional to,

p(US(t)|CS(t)) =∑

Cc p(US(t)|CS(t),c)p(CS(t)|c)p(c|ot)

∑Cc p(CS(t)|c)p(c|ot)

,

(4)where p(c|ot) is the posterior distribution across contextsgiven we have observed ot , in turn computed as,

p(c|ot) =∑x p(x|ot)p(x|c)p(c)

∑c ∑x p(x|ot)p(x|c)p(c). (5)

Changes in the environmental context can be easily in-corporated by marginalizing over S. Additionally, we canalso compute cumulative conditioning responses arising frommultiple stimuli, e.g. {CS1,CS2,S} in cue combination set-tings by marginalizing across the subset X of the cues in ques-

1504

0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

Trial

Co

nd

itio

ned

res

po

nse

p(US|CS)p(US|S)

0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

Trial

Co

nte

xt p

rob

abili

ty

US, CS, S

CS, SS

(A) (B)

Figure 1: Simulation illustrating the basic operation of our model. We assume that agents can estimate the likelihood ofseeing a stimulus worth responding to conditioned on the probability of being in particular situations. (A) Our Bayesian modelestimates conditional probabilities of US with respect to multiple environmental cues, giving us the traditional CR as a specialcase. (B) The principal component of our model is situation inference, which gates the accumulation of reinforcement evidence.

tion,

CR =X

∑x

p(US(t)|x)p(x) =X

∑x

p(US(t)|x)C

∑c

p(x|c)p(c|ot).

(6)Figure 1 shows the basic operation of our model as a sim-

ulation, with the measured conditioning response associatedwith both CS and S plotted across an acquisition-extinctionepisode in panel A and the corresponding latent context prob-abilities plotted in panel B. For the first 20 trials in this ex-periment, the agent is exposed to the unreinforced CS in theenvironmental condition S, resulting in a high context proba-bility for {CS, S}. Then, during acquisition between trials 20and 60, the context {US, CS, S} expectedly becomes promi-nent. Finally, the agent is exposed to the unreinforced CS in adifferent environmental condition, resulting in the dominanceof the context indexed by {CS}. Our simulation of CR (dot-ted line) shows the correct acquisition and extinction behaviorexpected of a model of classical conditioning.

ResultsOur theory makes accurate predictions for a large numberof conditioning-related effects. In the interests of succinct-ness, we restrict ourselves to reporting here 10 simulated ef-fect replications, selected for their prominence across multi-ple theoretical accounts of conditioning. No existing modelof conditioning can account for all 10 of the effects that weexplain below. In all the following figures, trial numbers arecounted from the last reinforced trial, i.e., the last reinforcedpresentation occurs at trial 0. For all subsequent trials, thesimulated agent sees unreinforced situation contexts in con-figurations determined by the corresponding test scenarios.Individual training and pre-exposure blocks, where specified,consisted of 5 trials in each of our simulations. Finally, sinceUS is a binary random variable, p(US|CS) = 0.5 will indicateabsence of conditioning if the training sequence is zero con-

tingency; all values lower than this can reflect absence of CRdepending on training conditions. Values higher than 0.5 willalways imply the existence of a CR.

2 4 6 8 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Trials

p(U

S|C

S 1)

CS1

CS1 + CS2

AB+ A− B−0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Training condition

Con

ditio

ning

resp

onse

(A) (B)

Figure 2: Stimulus generalization. Replicating (A) externalinhibition, where the trained CS is accompanied by a novelstimulus during testing and (B) non-linear (positive) pattern-ing, where individual elements A and B are presented un-reinforced, while the combined cue AB is reinforced duringtraining.

The first pair of effects, illustrated in Figure 2, can bothbe seen as instances of animals generalizing US-CS contin-gency information in interesting ways that simple Rescorla-Wagner (RW) models find hard to replicate. For example, theexample of non-linear patterning seen in 2(B) poses a ma-jor challenge for RW, since positive contingency of the USwith AB must necessarily transfer to both A and B by its pre-dictions. Instead, real animals, comparator-based accountsand our simulation all show CR in the presence of AB, butnot A or B, as shown in 2(A). Our model explains patterningthrough separate evidence accumulation for the situation con-texts triggered by the presence respectively of A,B and AB.External inhibition occurs when the presence of a novel stim-ulus during testing reduces CR, as shown in 2(A). This effectis, in fact, known since the earliest conditioning experiments,

1505

when Pavlov noticed the effects of extraneous noises on hisdogs’ CRs during testing. Our model explains this effect viaa reduction in the likelihood p(x|c) for an observation con-taining novel stimuli.

Another simple manipulation of the basic conditioning ex-periment involves pre-exposing a test group of subjects to theUS or CS before training begins. Learned irrelevance is aprominent effect that arises in such settings, often invokedby attention-based modelers, wherein prior exposure to USintermittently paired with CS retards subsequent acquisitionof CR for CS. Figure 3(A) illustrates our model’s replicationof this effect using two pre-exposure blocks containing to-gether five reinforced and five unreinforced presentations ofCS in random order, with the reduced sensitivity to trainingemerging from a flattened prior distribution p(c|o) of possiblesituations where US is expected. In other words, the modelanticipates that US will be present in a large range of situ-ations, the presence of CS being one of them. This dilutesthe probabilistic binding p(US|CS), leading to lower CR, andslower acquisition until p(c|o) becomes sufficiently peakedthrough repeated exposure to CS.

0 2 4 6 8 100

0.2

0.4

0.6

0.8

1

Trials

p(US|C

S)

NormalPre−exposed

0 2 4 6 8 100

0.2

0.4

0.6

0.8

1

Trials

p(US|C

S)

NormalPre−trained

(A) (B)

Figure 3: Pre-exposure effects. Replicating pre-exposure ef-fects like, (A) Learned irrelevance through random prior pre-sentation of US and CS leads to no initial CR (see trial 2)and slower acquisition of CR compared to normal condition-ing without pre-exposure to the random trials. (B) The Hall-Pearce effect, where pre-training with a weaker version of theUS retards subsequent acquisition with normal US.

Note that our explanation does not invoke notions ofsurprise and salience, like (Mackintosh, 1975) and otherattention-based accounts of this effect. The explanation forretarded CR acquisition arises entirely from our straightfor-ward Bayesian formalism. A similar resolution occurs forthe well-known Hall-Pearce effect, wherein pre-training witha weaker version of the US retards subsequent CR acquisi-tion with a stronger version. While our model does not per-mit explicit representations of stimulus strength, we modelthis effect by modifying our set membership computation inEquation 3 to permit partial matches. For our purposes, thismodification is restricted to defining p(x = weak US|c) =0.5,∀c s.t.US ∈ c and p(x = US|c) = 0.5,∀c s.t.weak US ∈c, following the intuition that a weak version of the sameUS will be judged similar to the US. With this assumption,our model replicates the Hall-Pearce effect using one pre-

exposure block presenting CS reinforced with weak US asdefined above, as illustrated in 3. As with learned irrelevance,no attention or surprise-related concepts are invoked.

When multiple conditioning stimuli are used, it turns outthat presenting one stimulus reinforced alone and unrein-forced when paired with another stimulus causes the secondstimulus to become a conditioned ‘inhibitor’, viz. its presencesuggests that US will likely not appear. Presenting both stim-ulus configurations for one trial block each leads to a repli-cation of conditioned inhibition in our simulation, as shownin 4(A). Bayesian inference provides an extremely natural ex-planation for this effect, since negative evidence for US avail-ability accumulates transparently in the {CS1,CS2} situationcontext in our model. Since CR is computed by marginaliz-ing across all such contexts, incorporating evidence from thiscontext reduces p(US|{CS1,CS2}), which, by the logic of ourmodel is the same as measuring CR in a summation test.

1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Trials

p(U

S|C

S)

CS1

{CS1, CS

2}

1 2 3 4 50.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

Trials

p(U

S|C

S2)

ControlCS

2−

CS2+

CS1+

(A) (B)

Figure 4: Conditioned inhibition. Replicating (A) the basicconditioned inhibition effect, where interleaved presentationof reinforced CS1 and an unreinforced combination of CS1and CS2 causes CS2 to be inhibited, as measured by sum-mation tests, and (B) variations thereof testing the effects ofsubsequent inflation and deflation of the inhibited CS.

As Gershman and Niv (2012) point out, there are multiplevariations of the basic conditioned inhibition result. For in-stance, once a stimulus has been trained as an inhibitor, sub-sequently reinforcing it attenuates the inhibition, while ex-tinguishing it has little impact on the conditioning response.This pattern is borne out in our simulations using one post-training block, as illustrated in 4(B). The y-intercept is the keyobservation in this figure, and shows clearly that the CS2−condition yields a similar CR to the baseline conditioned in-hibition setting, while CS2+ leads to increased CR at test trial1, though still inhibitory with respect to CS2. However, ourtheory fails, like Gershman’s model, to explain an observa-tion (Amundson, Wheeler, & Miller, 2005) that post-traininginflation of the conditioned excitor (CS1+) enhances condi-tioned inhibition of CS2. Our theory predicts the opposite re-sult, using positive evidence for US−CS1 binding to increasethe overall conditioning response, without reducing the rela-tive share of CS2.

Latent variable models also find it difficult to explain othercue competition based conditioning effects. The observ-able nature of our context variables renders our theory some-what more successful in explaining such data. Overshadow-

1506

ing is a classic conditioning effect, and has heretofore beenone of the most compelling pieces of evidence supportingvalue-based models of conditioning. In overshadowing, theCR for a stimulus trained jointly with another stimulus isweaker than the CR obtained when training with it alone.Rescorla-Wagner and other value-based models find it easyto explain this result as a sharing of value across multiplesites in the compound conditioning case. Our model ex-plains this effect, as illustrated in 5(A) via a simple condi-tionalization argument. Given the same number of trials ineach case, evidence for CS1 in the compound case has to beweighted by the observation likelihood of encountering CS1alone. Since this quantity is sub-unity by the design of theexperiment, overshadowing emerges naturally. Note that it isour model’s use of observation-indexed context variables thatpermits likelihood computation. Latent variable models usingmore abstract context variables cannot compute such quanti-ties, which makes it harder for them to explain cue competi-tion effects.

1 2 3 4 50.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Trials

p(US|C

S1)

CS1CS

2+

CS1+

1 2 3 4 50.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Trials

p(US|C

S2)

NormalBLUB

(A) (B)

Figure 5: Cue competition. Replicating effects like (A) over-shadowing, where compound conditioning with CS1 and CS2results in a weaker conditioning response than training witheither of the individual CS, and (B) the classic blocking ef-fect, where reinforcement with CS1 followed by reinforce-ment with {CS1,CS2} elicits no trained response for CS2 aswell as an unblocking effect, where blocking is attenuated bytesting in a new environmental context.

Figure 5(B) illustrates our model’s replication of Kamin’sclassic blocking effect, wherein it is found that reinforcingthe compound {CS1,CS2} after reinforcing CS1 yields notraining response for CS2. The existence of this effect mo-tivated the use of a maximum (saturation) level of trainingassociated with any US in the Rescorla-Wagner framework.It is also explained in attention-based frameworks by posit-ing that since CS1 already predicts US, the animal doesn’tneed to learn a new predictor CS2. Our model providesa novel inductive explanation. The observation likelihoodp(x =CS2|c = {CS1,CS2}) reduces the positive evidence forp(US|CS2), the prior encounters with the context containingjust CS1 lower the prior context probability p(c|o1:t) of in-ferring the context {CS1,CS2} itself. These sequential dilu-tions multiplicatively reduce the p(US|CS2) computation tothe value we see in 5(B). Also, our model replicates the un-blocking effect described by (Courville et al., 2006), whereinblocking is attentuated by testing in a new environmental con-

text. Here, adding a novel environmental stimulus reduces therelevance of the original CS1 trials in the context probabilitycalculation, resulting in a relatively higher value for contextscontaining CS2 than the blocking case.

The last pair of effects we consider have been been dif-ficult to explain in the past, particularly for latent variablemodels. The first - the partial reinforcement extinction effect(PREE) refers to the observation that CRs trained using in-termittent and variable schedules are larger in magnitude andtake longer to extinguish. As we show in 6(A), our model pre-dicts the second aspect of this effect, but not the first. In doingso, it joins the model of (Courville et al., 2006), who makeprecisely the same prediction. Our model explains this aspectof PREE following much the same outline as (Gershman &Niv, 2012) and (Courville et al., 2006). Intermittent observa-tion of the reinforcing context makes it harder for the induc-tive algorithm to judge when it is no longer in a reinforcingtrial block, thereby taking longer to extinguish the CR.

0 5 10 15 200

0.2

0.4

0.6

0.8

1

Trials

Rel

ativ

e co

nditi

onin

g re

spon

se

CompletePartial

1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Trials

p(U

S|C

S)

ControlLILI + OV

(A) (B)

Figure 6: Tricky effects. The situational context modelmakes interesting predictions for effects that have proved dif-ficult to explain such as (A) the partial reinforcement ex-tinction effect, where intermittent and variable reinforcementschedules retard future cue extinction. and (B) pre-exposureovershadowing, where unreinforced pre-exposure to CS1 ap-pears to eliminate the over-shadowing effect of subsequentcombined training with CS1 and CS2.

Courville et al. (2006) also describe preexposure overshad-owing (Experiment 3 in their paper), where unreinforcedpreexposure to CS1 eliminates the overshadowing effect ofsubsequent compound training with {CS1,CS2}. This is,on the surface, a surprising result, since it is known that(i) pre-exposure to a stimulus tends to retard future acqui-sition (latent inhibition), and (ii) pairing two stimuli alsoleads to slower learning (overshadowing). Blaisdell and col-leagues (Blaisdell, Bristol, Gunther, & Miller, 1998) demon-strated that combining both these conditions actually atten-uates the learning deficit. In fact, as a mark of its intri-cacy, (Blaisdell et al., 1998) set it up as a challenge for non-comparator accounts of conditioning. Our model replicatesthis effect, as shown in 6(B), and explains it thus: prior expo-sure to CS1 tilts the context probability p(c|o) in favor of con-texts containing only CS1 as opposed to the pair {CS1,CS2}.This causes evidence for the compound to disproportionatelyraise p(US|CS1) above the overshadowing baseline.

1507

Latent causes

Stimuli

Situation contexts

Latent causes

Situation contexts

Courville, Daw & Touretzky Gershman, Niv & Blei Srivastava & Schrater

Stimuli

Figure 7: A comparison of the generative models underpinning our theory and its closest relatives - Courville’s latent causemodel and Gershman’s infinite capacity latent cause model. The key difference is the observability of the context variables inour theory which permits more granular likelihood computations and a richer set of effect descriptions.

DiscussionThe principal source of variation between the models inFigure 7 is in the interpretation of the context variables.Courville et al. (2006) assume that the latent variables aregenerative models that the animal thinks could explain the ob-servations it sees. Gershman and Niv (2012) assume that la-tent variables are clusters of observations, imputing a specificgenerative model (Dirichlet process mixtures) to the cluster-ing process. Our interpretation indexes contexts using obser-vation co-occurence directly. The consequent transparencyof representation allows our model to reproduce the predic-tions of its competitors, and predict some effects they cannot.The main contribution of this work is a streamlining of theassumptions underlying the inferential view of conditioning,thereby improving its explanatory power and testability.

An important limitation of our model is that it is event-based, rather than time-based. While the time elapsed be-tween events is certainly important for many kinds of learn-ing, including reinforcement tasks that have explicit time de-pendence on the delivery and availability food or reinforcers,our methods have focused on another ubiquitous type of de-cision task, where the options selected are freely availablewithin a context, and the context determines what shouldbe selected. Such tasks are likely to be well-described bycontext-dependent conditioning, wherein ventromedial pre-frontal cortex and hippocampus loops have been neurophysi-ologically implicated in context-specific retrieval of value in-formation (Kalisch et al., 2006).

In a broad sense, value-based theories of conditioning de-scribe a world where understanding is cheap and decisions aredifficult. Such a description is apt for artificial control sys-tems, the study of which, not coincidentally, gives us muchof the mathematics that governs value-based learning theo-ries. It is not, however, a fitting description of the way well-adapted animals engage with their natural environments. Theinference-based view of conditioning gives a better descrip-tion of such a world - where understanding is difficult, andaction selection, once the situation context is understood, iseasy. The success of our simple model in explaining a varietyof conditioning effects further strengthens this belief.

ReferencesAmundson, J. C., Wheeler, D. S., & Miller, R. R. (2005).

Enhancement of pavlovian conditioned inhibition achievedby posttraining inflation of the training excitor. Learningand motivation, 36(3), 331–352.

Blaisdell, A. P., Bristol, A. S., Gunther, L. M., & Miller, R. R.(1998). Overshadowing and latent inhibition counteracteach other: Support for the comparator hypothesis. Journalof Experimental Psychology: Animal Behavior Processes,24(3), 335.

Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006).Bayesian theories of conditioning in a changing world.Trends in cognitive sciences, 10(7), 294–300.

Gershman, S. J., & Niv, Y. (2012). Exploring a latent causetheory of classical conditioning. Learning & behavior,40(3), 255–268.

Hasher, L., & Zacks, R. T. (1984). Automatic processing offundamental information: the case of frequency of occur-rence. American Psychologist, 39(12), 1372.

Kalisch, R., Korenfeld, E., Stephan, K. E., Weiskopf, N.,Seymour, B., & Dolan, R. J. (2006). Context-dependenthuman extinction memory is mediated by a ventromedialprefrontal and hippocampal network. The Journal of neu-roscience, 26(37), 9503–9511.

Knill, D. C., & Pouget, A. (2004). The bayesian brain:the role of uncertainty in neural coding and computation.TRENDS in Neurosciences, 27(12), 712–719.

Mackintosh, N. J. (1975). A theory of attention: Variationsin the associability of stimuli with reinforcement. Psycho-logical Review, 82(4), 276.

Pavlov, I. P. (1927). Conditioned reflexes. DoverPublications.com.

Rescorla, R. A., & Wagner, A. R. (1972). A theory of pavlo-vian conditioning: Variations in the effectiveness of rein-forcement and nonreinforcement. Classical conditioningII: Current research and theory, 64–99.

Stout, S. C., & Miller, R. R. (2007). Sometimes-competingretrieval (socr): A formalization of the comparator hypoth-esis. Psychological Review, 114(3), 759.

1508