DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING ...

DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS

BRIAN LAU AND PAUL W. GLIMCHER

NEW YORK UNIVERSITY

We studied the choice behavior of 2 monkeys in a discrete-trial task with reinforcement contingenciessimilar to those Herrnstein (1961) used when he described the matching law. In each session, themonkeys experienced blocks of discrete trials at different relative-reinforcer frequencies or magnitudeswith unsignalled transitions between the blocks. Steady-state data following adjustment to eachtransition were well characterized by the generalized matching law; response ratios undermatchedreinforcer frequency ratios but matched reinforcer magnitude ratios. We modelled response-by-response behavior with linear models that used past reinforcers as well as past choices to predict themonkeys’ choices on each trial. We found that more recently obtained reinforcers more stronglyinfluenced choice behavior. Perhaps surprisingly, we also found that the monkeys’ actions wereinfluenced by the pattern of their own past choices. It was necessary to incorporate both past reinforcersand past choices in order to accurately capture steady-state behavior as well as the fluctuations duringblock transitions and the response-by-response patterns of behavior. Our results suggest that simplereinforcement learning models must account for the effects of past choices to accurately characterizebehavior in this task, and that models with these properties provide a conceptual tool for studying howboth past reinforcers and past choices are integrated by the neural systems that generate behavior.

Key words: choice, matching law, dynamics, model, eye movement, monkey

_______________________________________________________________________________

Much of our understanding of choicebehavior comes from the rich body of researchusing concurrent schedules of reinforcementlike those Herrnstein used to describe thematching law (Herrnstein, 1961). Neuroscien-tists studying the biological mechanisms ofchoice behavior are poised to take advantageof the techniques and results that have beendeveloped from studies on matching behavior(reviewed in Davison & McCarthy, 1988; deVilliers, 1977; B. Williams, 1988). However,monkeys are commonly used as subjects inneuroscientific research, but they are rarelyused as subjects in matching experiments, andthe behavioral research that has been donewith monkeys (Anderson, Velkey, & Woolver-ton, 2002; Iglauer & Woods, 1974) differs in

a number of procedural details from thebehavioral methods used in neuroscience. Inneurophysiological experiments, monkeys arehead-restrained to allow for stable electrophys-iological recordings, eye movements ratherthan arm movements are often used as a re-sponse measure, individual responses occur indiscrete-trials to allow for precise control oftiming, and water rather than food is used asa reinforcer to minimize recording artifactscaused by mouth movements. And perhapsmost importantly, the reinforcement contin-gencies differ from the concurrent variable-interval (VI) schedules used to study matchingbehavior (but see Sugrue, Corrado, & News-ome, 2004). These differences make it difficultto relate monkey choice behavior as describedby neuroscientists to the existing literature onthe matching law.

One of our goals in the present study was torigorously characterize the behavior of mon-keys using apparatus typical of neurophysio-logical experiments while they performeda repeated choice task with reinforcementcontingencies similar to the concurrent VI VIschedules used to study matching behavior.We used a discrete-trial task where theprobability of reinforcement for choosinga particular alternative grows with the numberof trials spent not choosing that alternative,similar to the exponential growth obtained

We thank Michael Davison, Nathaniel Daw, PeterDayan, Mehrdad Jazayeri, and Aldo Rustichini for helpfuldiscussions. Portions of this paper were presented at the2004 meeting of Society for the Quantitative Analyses ofBehavior and at Computational and Systems Neuroscience2004. Supported by a NDSEG fellowship to Brian Lau andNEI EY010536 to Paul W. Glimcher.

All experimental procedures were approved by the NewYork University Institutional Animal Care and UseCommittee, and performed in compliance with the PublicHealth Service’s Guide for the Care and Use of Animals.

Address correspondence to Brian Lau, Center forNeural Science, New York University, 4 Washington Place,Room 809, New York, New York 10003 (e-mail: [email protected]).

doi: 10.1901/jeab.2005.110-04

JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR 2005, 84, 555–579 NUMBER 3 (NOVEMBER)

555

with constant-probability VI schedules (e.g.,Houston & McNamara, 1981; Staddon, Hin-son, & Kram, 1981). And like concurrent VI VIschedules, this method of arranging reinforc-ers renders exclusive choice of an alternativesuboptimal for maximizing reinforcer ratebecause at some point the probability ofreinforcement of the unchosen alternativeexceeds that of the chosen alternative. In fact,for tasks like ours, maximizing reinforcer rateis closely approximated when subjects matchthe proportion of choices allocated to analternative to the proportion of reinforcersreceived from choosing it (Houston & McNa-mara, 1981; Staddon et al., 1981). This also istrue of concurrent VI VI schedules (Staddon &Motheral, 1978), and allows us to compare ourresults with previous work on the matchinglaw.

Classic descriptions of choice relate behav-ioral output, typically measured as the long-term average ratio of choices, to averagereinforcer input, typically measured as thelong-term average ratio of reinforcers. Thestrict matching law equates these two quanti-ties; the relative number of choices matchesthe relative reinforcer frequency (Herrnstein,1961). This description has been extended toinclude other reinforcer attributes such asmagnitude, delay, and quality (reviewed inDavison & McCarthy, 1988; de Villiers, 1977; B.Williams, 1988), and generalized to accountfor deviations from strict matching (Baum,1974). The generalized matching law forreinforcer frequency and magnitude assumesthat these attributes have independent effects(Baum & Rachlin, 1969), but allows fordifferential levels of control by each variable;

C1

C2~c

R1

R2

� �a M1

M2

� �b

, ð1Þ

where C, R, and M denote the number ofresponses, number of reinforcers, and re-inforcer magnitude for each alternative, re-spectively. The coefficients a and b relate theratio of responses to reinforcer frequency andmagnitude, and are interpreted as sensitivity toeach variable, whereas c is a constant biastowards one alternative not related to re-inforcer frequency or magnitude.

The generalized matching law describes anenormous amount of data collected frommany species (de Villiers, 1977), and allows

one to predict average choice behavior underdifferent conditions, such as for arbitrarycombinations of reinforcer frequency andmagnitude. However, the generalized match-ing law does not specify how animals producematching behavior at a response-by-responselevel. As such, the generalized matching law isless useful for making neurophysiologicalpredictions because it does not constrain howneural activity should vary on a response-by-response basis. For neurobiological studies,a model of performance that specifies thecomputations performed by animals whenallocating behavior would be of tremendousvalue because interpreting neural data oftenrequires understanding how choices are pro-duced. Ideally, such a model would furnishresponse-by-response estimates of the underly-ing decision variables (e.g., reinforcer rate)that predict the animal’s choices. Measure-ments of brain activity then can be correlatedwith these theoretical variables to determinewhether they predict brain activity as well aschoice (e.g., O’Doherty et al., 2004; Platt &Glimcher, 1999; Sugrue et al., 2004).

Within psychology, a number of theoreticalmodels have been formulated to explainmatching behavior (B. Williams, 1988). Someof these models make predictions at theresponse-by-response level (e.g., momentarymaximizing, Shimp, 1966), and considerableeffort has been devoted to dissecting the localstructure of choice to select amongst thesedifferent response-by-response models. Manyof the analytical methods used to understandhow local variations in behavior lead tomatching can be broadly categorized as re-lating local variations in behavior to (a) localvariations in reinforcer history or (b) localvariations in behavioral history itself. Examplesof the first class include transfer functionformulations used by Palya and colleagues(Palya, Walter, Kessel, & Lucke, 1996, 2002;see McDowell, Bass, & Kessel, 1992, fora theoretical exposition in a behavioral con-text) to predict response rate using weightedsums of past reinforcers. Related techniqueshave been applied to predict choice allocationin concurrent schedules on short time scales(response-by-response, Sugrue et al., 2004) aswell as longer time scales (within and betweensessions, Grace, Bragason, & McLean, 1999;Hunter & Davison, 1985; Mark & Gallistel,1994). An example of the second class of

556 BRIAN LAU and PAUL W. GLIMCHER

techniques is estimating the probability ofa particular choice conditional on a specificsequence of past choices, which can revealthe types of strategies animals use underdifferent reinforcement conditions (e.g., Hey-man, 1979; Nevin, 1969; Shimp, 1966; Silber-berg, Hamilton, Ziriax, & Casey, 1978). Al-though it is conceptually useful to separate theeffects of reinforcers and the potentially in-trinsic response-by-response patterning of be-havior (e.g., tendencies to generate particularsequences of behavior), behavior generallyrepresents some combination of these two(Davison, 2004; Heyman, 1979; Palya, 1992).This suggests that analyses of reinforcer effectsor sequential choice effects in isolation maynot have a straightforward interpretation.Consider a situation in which an animal strictlyalternates between two choice options. Re-inforcement only is available from one alter-native, and when received, causes the animalto produce one extra choice to it after whichstrict alternation is continued. Focusing solelyon the relation between choice behavior andpast reinforcers might lead to the incorrectinterpretation that reinforcement changeschoice behavior in an oscillatory manner.

These considerations led us to formulatea response-by-response model that treats theeffects due to past reinforcers and choices asseparable processes. We used this statisticalmodel to predict individual choices based onweighted combinations of recently obtainedreinforcers as well as previous behavior (c.f.,Davison & Hunter, 1979), which allowedchoice predictions to be influenced both bywhich alternatives had recently yielded re-inforcers, as well as by sequential patterns inchoice behavior.

The present results help test existing modelsof choice behavior. Many of the more plausi-ble—in the sense that animals may actuallyimplement them—theoretical models incor-porate some form of reinforcer rate estimationusing linear weightings of past reinforcers(e.g., Killeen, 1981; Luce, 1959). Recently,models using reinforcer rate estimation havebeen used to make predictions about neuralactivity. Single neurons in a number of corticalareas have activity correlated with reinforcerfrequency (Barraclough, Conroy, & Lee, 2004;Platt & Glimcher, 1999; Sugrue et al., 2004)and magnitude (Platt & Glimcher, 1999).These and related studies (Glimcher, 2002;

Montague & Berns, 2002; Schultz, 2004) arebeginning to lay the groundwork for a detailedunderstanding of the mechanisms underlyingchoice behavior, and highlight the importanceof behavioral models in forging a bridgebetween behavior and brain. We have foundthat local linear weightings of past reinforcerspredict the choice behavior of monkeysperforming a matching task, with weightssimilar to those suggested by reinforcer rateestimation models, but we extend this to showthat accounting for recent choices also isrequired for accurate predictions by modelsof this type. The present results suggest thatbehavioral models incorporating dependenceon both past reinforcers and choices mayprovide more accurate predictions, which willbe essential for correlation with neural activity.

METHOD

Subjects

Two male rhesus monkeys (Macaca mulatta)were used as subjects (Monkey B and MonkeyH, 10.5 kg and 11.5 kg respectively at the startof the experiments). Monkey H had been usedpreviously in another experiment studying eyemovements. Monkey B was naive at the start ofthe experiments described here. Both mon-keys experienced a similar sequence of train-ing (described below), and both were naive tothe choice task used in this experiment.

Apparatus

Prior to behavioral training, each animal wasimplanted with a head restraint prosthesis anda scleral eye coil to allow for the maintenanceof stable head position and recording of eyeposition. Surgical procedures were performedusing standard aseptic techniques under iso-flurane inhalant anaesthesia (e.g., Platt &Glimcher, 1997). Analgesia and antibioticswere administered during surgery and contin-ued for 3 days postoperatively.

Training sessions were conducted in a dimlylit sound-attenuated room. The monkeys sat inan enclosed plexiglass primate chair (28 cm by48 cm) that permitted arm and leg move-ments. The monkeys were head-restrainedusing the implanted prosthesis. Body move-ments were monitored from a separate roomwith a closed-circuit infrared camera. Eyemovements were monitored and recorded

LINEAR MODELS FOR REPEATED CHOICE 557

using the scleral search coil technique (Judge,Richmond, & Chu, 1980) with a sampling rateof 500 Hz.

Visual stimuli were generated using an arrayof light-emitting diodes (LEDs) positioned145 cm from the subject’s eyes. The arraycontained 567 LEDs (21 3 27, 2u spacing), andeach LED subtended about 0.25u of visualangle.

Procedure

Training began after a postoperative re-covery period of 6 weeks. Water delivered intoa monkey’s mouth via a sipper tube was used asthe reinforcer. The monkeys were trained onweekdays, and obtained the bulk of their wateron these days during the 2 to 4 hr trainingsessions. The monkeys were not trained on theweekends, during which ad libitum water wasprovided. To ensure that the monkeys wereadequately hydrated, we monitored the mon-keys’ weights daily and the University veteri-narian periodically assessed the condition ofthe animals.

Initial training consisted of habituating themonkeys to being head-restrained in theprimate chair. They were brought up fromtheir home cages daily, and sitting quietly inthe primate chair was reinforced with water.After 1 to 2 weeks of chair training, themonkeys were trained to shift their point ofgaze to visual targets and to hold fixation usingwater reinforcement. We used manually con-trolled reinforcement to shape immediate andrapid movements to briefly illuminated LEDs,

and incrementally delayed reinforcement toshape prolonged fixation. Once the monkeyscould shift gaze rapidly to targets presentedanywhere in the LED array and maintainfixation for 1 to 2 s on that LED, we placedreinforcement under computer control. Themonkeys then were trained to shift gaze toeccentric visual targets from a central fixationpoint. When the monkeys were performingthese tasks reliably, we began training ona choice task.

The data reported in this paper werecollected while the monkeys performed thetwo-alternative choice task shown in Figure 1.Each trial started with a 500 ms 500 Hz tone,after which the monkey was given 700 ms toalign their gaze within 3u of a yellow LED inthe center of the visual field. After maintainingfixation for 400 ms, two peripheral LEDs (onered and one green) were illuminated on eitherside of the centrally located fixation point.These peripheral LEDs were positioned anequal distance from the fixation point; this wasoccasionally varied by a few degrees fromsession to session, but the distance was, onaverage, 15u of visual angle. One second later,the central fixation point disappeared, cueingthe monkey to choose one of the peripheralLEDs by shifting gaze to within 4u of itslocation and fixating for 600 ms. If a reinforcerhad been scheduled for the target chosen, itwas delivered 200 ms after the eye movementwas completed. The trial durations were thesame whether or not the animal receivedreinforcement. Each trial lasted 3.2 to 3.65 s,

Fig. 1. Timeline and spatial arrangement of the two-alternative choice task.


and only one choice could be made on eachtrial. Trials were separated by a 2-s intertrialinterval (ITI) beginning at the end of the timea reinforcer would have been delivered. NoLEDs were illuminated during the ITI. Therange in trial durations comes from thevariability in the amount of time taken by themonkeys to select and execute their eyemovements. Relative to the duration of thetrial, this time was short (mean reaction time187 ms with a standard deviation of 71 ms).

A trial was aborted if the monkey failed toalign its gaze within the required distancesfrom the fixation or choice targets, or if an eyemovement was made to one of the choicetargets before the fixation point was extin-guished. When an abort was detected, all theLEDs were extinguished immediately, no re-inforcers were delivered, and the trial wasrestarted after 3 s. The monkeys rarely abortedtrials (7% and 6% of trials for Monkey H andMonkey B, respectively), and these trials wereexcluded from our analyses.

Reinforcers were arranged by flipping a sep-arate biased coin for each alternative usinga computer random-number generator; if‘‘heads’’ came up for a particular alternative,a reinforcer was armed, or scheduled, for thatalternative. Reinforcers were scheduled usingindependent arming probabilities for eachalternative, meaning that on any trial bothalternatives, neither alternative, or only onealternative might be armed to deliver a re-inforcer. If a reinforcer was scheduled for thealternative the monkey did not choose, itremained available until that alternative wasnext chosen (however, no information re-garding held reinforcers was revealed to thesubjects). For example, if a subject chose theleft alternative when both alternatives werearmed to deliver reinforcers, he would receivethe reinforcer scheduled for the left alterna-tive while the reinforcer for the right alterna-tive would be held until the next time thesubject chose it, which could be any number oftrials into the future. A changeover delay orchangeover response was not used; reinforcerswere not delayed or withheld for an extraresponse when subjects chose to switch fromselecting one alternative to the other. Thismethod of arranging reinforcers in a discrete-trial choice task has been referred to as‘‘dual assignment with hold’’ (Staddon et al.,1981).

In one set of sessions, we held the magni-tude of reinforcement obtained from choosingeither alternative equal while varying relativereinforcer frequency within sessions. We usedarming probabilities that summed to about 0.3(see Table 1 for precise values), and in eachsingle session, the monkeys performed a seriesof trials under four different arming probabil-ity ratios (in Condition 1 for example, theratios were 6:1, 3:1, 1:3, and 1:6). We switchedbetween these ratios in an unsignalled manneras described below.

In another set of sessions, we held theprogrammed arming probabilities constantand equal while varying the relative reinforcermagnitudes within sessions. The amount ofwater delivered was controlled by varying theamount of time a solenoid inline with thewater spout was held open. In each session, themonkeys performed a series of trials underfour different magnitude ratios (in Condition12 for example, the ratios were 3:1, 3:2, 2:3,1:3), where the programmed arming probabil-ity for both alternatives was 0.15. We switchedbetween these ratios in an unsignalled manneras described below.

Note that for the arming probabilities used,the subjects did not receive reinforcement onevery trial; the monkeys received, on average,a reinforcer every three to four trials. However,aside from whether or not a reinforcer wasdelivered, the timing and appearance of eachtrial was identical to the subjects.

In each session, the monkeys performeda series of choice trials consisting of blocks oftrials at different relative reinforcer frequen-cies or different relative reinforcer magni-tudes. For example, in a frequency condition,a monkey might perform 124 trials where theright and left alternatives had an armingprobability of 0.21 and 0.07, respectively (a3:1 ratio) then 102 trials at a 1:6 ratio followedby 185 trials at a 3:1 ratio. Transitions betweenblocks of trials with different ratios wereunsignalled, and the monkeys had to learnby experience which alternative had thehigher frequency or magnitude of reinforce-ment. When blocks were switched, the richeralternative always changed spatial location, butits degree of richness was variable; the twopossible ratios to switch to were chosen withequal probability. For example, a 3:1 or 6:1ratio was followed by a 1:6 or a 1:3 ratio withequal probability. There were minor variations


in the ratios (Table 1), but we used thesame method for block transitions in allfrequency and magnitude conditions as de-scribed next.

We randomized the number of trials in eachblock to discourage any control over behaviorby the number of trials completed in a block.The number of trials in a block for allconditions was 100 trials plus a randomnumber of trials drawn from a geometricdistribution with a mean of 30 trials. Themean number of trials per block observed was

121 trials for Monkey H and 125 trials forMonkey B, after excluding aborted trials.

The data analyzed were 67,827 completedtrials from Monkey H and 47,160 completedtrials from Monkey B. All the data reportedwere collected after at least 3 months oftraining on the choice task.

RESULTS

We present our results in two sections. Firstwe describe the steady-state behavior at the

Table 1

Experimental conditions for each monkey. The arming probability and magnitude ofreinforcement are listed for each alternative. The magnitude of reinforcement is in units ofmilliliters.

Monkey H Monkey B

Armingprobabilities Magnitudes

Number ofblocks

Armingprobabilities Magnitudes

Number ofblocks

Frequencycondition

Frequencycondition

1 0.24/0.04 0.35/0.35 16 5 0.25/0.05 0.4/0.4 50.21/0.07 0.35/0.35 12 0.22/0.08 0.4/0.4 90.07/0.21 0.35/0.35 11 0.08/0.22 0.4/0.4 70.04/0.24 0.35/0.35 19 0.05/0.25 0.4/0.4 5

2 0.24/0.04 0.4/0.4 5 6 0.25/0.05 0.5/0.5 180.21/0.07 0.4/0.4 8 0.22/0.08 0.5/0.5 150.07/0.21 0.4/0.4 8 0.08/0.22 0.5/0.5 110.04/0.24 0.4/0.4 4 0.05/0.25 0.5/0.5 18

3 0.24/0.04 0.45/0.45 7 7 0.29/0.04 0.5/0.5 120.21/0.07 0.45/0.45 10 0.24/0.09 0.5/0.5 150.07/0.21 0.45/0.45 5 0.09/0.24 0.5/0.5 130.04/0.24 0.45/0.45 10 0.04/0.29 0.5/0.5 17

4 0.24/0.04 0.5/0.5 10 8 0.283/0.047 0.55/0.55 90.21/0.07 0.5/0.5 13 0.248/0.082 0.55/0.55 80.07/0.21 0.5/0.5 10 0.082/0.248 0.55/0.55 100.04/0.24 0.5/0.5 12 0.047/0.283 0.55/0.55 3

5 0.25/0.05 0.4/0.4 9 9 0.283/0.047 0.6/0.6 30.22/0.08 0.4/0.4 3 0.248/0.082 0.6/0.6 50.08/0.22 0.4/0.4 7 0.082/0.248 0.6/0.6 50.05/0.25 0.4/0.4 8 0.047/0.283 0.6/0.6 3

6 0.25/0.05 0.5/0.5 210.22/0.08 0.5/0.5 220.08/0.22 0.5/0.5 170.05/0.25 0.5/0.5 18

Magnitudecondition

Magnitudecondition

10 0.15/0.15 0.6/0.2 12 10 0.15/0.15 0.6/0.2 70.15/0.15 0.5/0.3 10 0.15/0.15 0.5/0.3 90.15/0.15 0.3/0.5 12 0.15/0.15 0.3/0.5 100.15/0.15 0.2/0.6 10 0.15/0.15 0.2/0.6 2

11 0.15/0.15 0.6/0.2 42 11 0.15/0.15 0.6/0.2 200.15/0.15 0.48/0.32 38 0.15/0.15 0.48/0.32 190.15/0.15 0.32/0.48 36 0.15/0.15 0.32/0.48 210.15/0.15 0.2/0.6 33 0.15/0.15 0.2/0.6 17

12 0.15/0.15 0.75/0.25 29 12 0.15/0.15 0.75/0.25 180.15/0.15 0.6/0.4 27 0.15/0.15 0.6/0.4 210.15/0.15 0.4/0.6 27 0.15/0.15 0.4/0.6 210.15/0.15 0.25/0.75 20 0.15/0.15 0.25/0.75 20


end of each block in terms of the generalizedmatching law. This establishes the behavioralendpoint that we seek to explain using re-sponse-by-response models in the subsequentsection.

Steady-State Analysis

Figure 2 shows examples of responses toblock transitions for each monkey. The curvesshow the probability of choosing the alterna-tive that was richer following an unsignalledblock transition, aligned on the trial at whichthe transition occurred. These probabilitieswere computed by averaging choices for trials

preceding and following a block transition.For example, the thick line in the upper leftpanel is an average of all transitions that endedwith an arming probability ratio of 24:4.Likewise, the thin lines in the lower panelsare averages of all transitions that ended witha magnitude ratio of 3:2. The curves in Figure 2are similar prior to the transition because bothare averages of the two possible pretransitionratios (see Methods). There was a rapidacquisition period following the transition,and the curves diverged for the two possibleposttransition ratios. We quantified the speedof acquisition using the number of trials it

Fig. 2. Example choice data aligned on the trial (dashed line) that a transition between ratios occurred. The upperpanels are data from conditions (Table 1) where reinforcer frequency was manipulated, and the lower panels are datafrom conditions where reinforcer magnitudes were manipulated. The data were compiled with respect to the alternativethat was richer following the transition, and averaged separately for the two possible programmed posttransition ratios(pretransition ratios were on average ,1:4 for the top panels, and ,1:2 for the bottom panels). The data were smoothedusing a five-point moving average.


took the monkeys to reach a 50% rate ofchoosing the richer alternative followinga block transition. This was 11.4 trials forMonkey H and 10.42 trials for Monkey Baveraged over all the conditions where re-inforcer frequency was manipulated. Therewas no difference between the acquisitiontimes for the two possible posttransition ratiosin each condition (,3:1 and ,6:1). Theacquisition time was 11.56 trials for MonkeyH and 10.86 trials for Monkey B averagedover all the conditions where reinforcermagnitude was manipulated. However, forthe reinforcer magnitude conditions, therewas a difference between these times thatdepended on the posttransition reinforcermagnitude ratio; transitions to a 3:1 ratiowere faster (9.84 trials for Monkey H and8.55 trials for Monkey B) than transitions toa 3:2 ratio (13.27 trials for Monkey H and13.17 trials for Monkey B).

Figure 2 shows that there was a periodfollowing acquisition where the animals’ be-havior fluctuated about a steady-state. Wecharacterized the differences in steady-statepreference by analyzing the last 65 trials ofeach block. We chose this number based onplots like Figure 2, which show that behaviorhad, on average, reached a steady-state by thispoint in each block.

Figure 3 shows log ratio responses fromexample reinforcer frequency and reinforcermagnitude conditions. We fit the data for eachcondition with the logarithmic form of thegeneralized matching law,

logC1

C2

� �~alog

R1

R2

� �zblog

M1

M2

� �zlogc, ð2Þ

using least-squares regression. Equation 2 wasused without the magnitude term for theconditions where we manipulated reinforcerfrequency, whereas for the magnitude condi-tions, we included the term for reinforcerfrequency because the reinforcement sched-ules did not enforce precise equality betweenthe obtained reinforcer frequencies from thetwo alternatives. The estimated parametersfrom these fits are listed in Table 2. Thecoefficients and quality of fits were similar forthe 2 monkeys, consistent with the examplesshown in Figure 3. Both animals showedgreater sensitivity to reinforcer magnitude (b)than reinforcer frequency (a).

Figure 4 shows log response ratios asa function of log obtained reinforcer frequen-cy and log reinforcer magnitude ratios for allour data. The plane through the data repre-sents a least-squares fit of Equation 2, exclud-ing data in which no reinforcers were obtainedfrom one of the alternatives (six of 551 blocksfor Monkey H, 10 of 376 blocks for Monkey B).The generalized matching law provided goodfits to the data, accounting for 90% of thevariance in the data for both monkeys.

We assessed the degree to which our datadeviated from the generalized matching law bychecking for a dependence of the observedbehavior on nonlinear transformations ofrelative reinforcer frequency and relative re-inforcer magnitude not captured by Equation2. We did this by adding polynomial covariatesto Equation 2; for the reinforcer frequencyconditions, we fit additional coefficients forlog reinforcer frequency ratio raised to thesecond and third powers, whereas for thereinforcer magnitude conditions, we fit addi-tional coefficients for log reinforcer frequencyand log reinforcer magnitude ratios raised tothe second and third powers. Including thesecovariates accounted for quadratic and cubicdeviations in the data, but only increased theaverage percentage of variance explained by1.8% (range, 0.31 to 3.81%) and 1.5% (range,0.13 to 2.63%) for the reinforcer frequencyand reinforcer magnitude conditions, respec-tively. This indicates that the generalizedmatching law is a reasonable description ofour data, even though it typically is used todescribe data after many sessions, rather thantrials, of training under the same reinforcerconditions.

Although the generalized matching law fitsthe steady-state behavior that we measured, itonly describes average choice. It is silentregarding the sequences of choices underlyingthese averages. At one extreme, we mighthypothesize that the response-by-response pro-cess that achieves steady-state behavior oper-ates is purely deterministic. At the otherextreme, we might hypothesize that behavioris allocated according to a purely probabilisticstrategy. We examined how probabilistic themonkeys were by studying how runlengths, thenumber of consecutive choices to a particularalternative, vary with the degree of preference,as measured by the relative responses fromeach block. If the monkeys chose in a purely


probabilistic manner by flipping a weightedcoin on each trial, with the weight dictated bythe degree of preference (e.g., a 2:1 prefer-ence gives a two-thirds probability of choosingthe preferred alternative on each trial), therunlengths would be distributed geometricallywith a mean equal to the reciprocal of the coinweight. This is a useful baseline model becausethe assumption of independence betweentrials that defines this model means that thedegree of weighting of the coin fully specifies

the runlength distributions (e.g., Houston &Sumida, 1987).

Figure 5 shows runlength as a function ofpreference, with these variables measuredfrom the last 65 trials in each block. Run-lengths were measured for both alternatives;for example, the sequence of choicesRLLLLLR from a block where L was preferredwould contribute a runlength of five to therunlengths for the rich, or preferred, alterna-tive and a runlength of one to the runlengths

Fig. 3. Log choice ratios (right over left) from individual conditions (see Table 1 for ratios used in each condition) asa function of obtained log reinforcer frequency ratios (upper panels) or log reinforcer magnitude ratios (lower panels).Each point was obtained by averaging the number of choices and reinforcers from the last 65 trials of a block. The linesrepresent least-squares fits of the generalized matching law, the coefficients of which are listed in Table 2. The solidsymbols plotted at the extremes of the abscissa (for Monkey B) represent blocks where no reinforcers were obtained fromone of the alternatives. For the magnitude conditions, the effect due to reinforcer frequency has been subtracted fromthe log choice ratios.


for the lean, or nonpreferred, alternative.Because neither animal exhibited a significantspatial bias, runlengths were compiled for thepreferred and nonpreferred alternatives irre-spective of spatial location. The points havebeen jittered slightly along the abscissa toreveal data that otherwise would have stackedon top of each other. Almost as soon as one ofthe alternatives was preferred, the runlengthsfor the nonpreferred alternative became very

short, typically one trial (mean across allpreferences was 1.05 trials with a standarddeviation of 0.09).

Figure 5 also shows the predicted averagerunlengths under the probabilistic strategydescribed above. The actual runlength datasystematically deviate from and hence rejectthis model, implying that average preferenceas specified by the generalized matching law isinsufficient to specify the average runlength

Table 2

Fits of the generalized matching law (Equation 2) to the steady-state data from the end of eachblock. The condition labels in the first column correspond to those in Table 1.

Monkey H Monkey B

log c (SE) a (SE) b (SE) R2 log c (SE) a (SE) b (SE) R2

Frequency condition1 0.06 (0.06) 0.51 (0.03) — 0.882 0.10 (0.09) 0.50 (0.04) — 0.893 20.08 (0.08) 0.52 (0.03) — 0.914 0.16 (0.07) 0.44 (0.03) — 0.875 0.00 (0.07) 0.57 (0.03) — 0.93 20.01 (0.09) 0.51 (0.04) — 0.886 0.05 (0.05) 0.54 (0.02) — 0.89 20.10 (0.07) 0.56 (0.03) — 0.867 0.00 (0.05) 0.58 (0.02) — 0.938 20.01 (0.08) 0.49 (0.03) — 0.899 20.10 (0.08) 0.56 (0.04) — 0.95

pooled 0.05 (0.03) 0.51 (0.01) — 0.88 20.04 (0.03) 0.54 (0.01) — 0.89

Magnitude condition10 0.08 (0.05) 0.31 (0.06) 1.02 (0.04) 0.96 20.15 (0.11) 0.43 (0.16) 1.31 (0.11) 0.9211 0.05 (0.03) 0.34 (0.05) 1.04 (0.03) 0.92 20.18 (0.05) 0.49 (0.07) 1.34 (0.05) 0.9512 0.08 (0.04) 0.31 (0.06) 0.94 (0.04) 0.90 20.05 (0.06) 0.56 (0.08) 1.08 (0.06) 0.88

pooled 0.06 (0.02) 0.32 (0.03) 1.00 (0.02) 0.92 20.11 (0.03) 0.53 (0.05) 1.22 (0.04) 0.91

Fig. 4. Log choice ratios (right over left) as a function of obtained log reinforcer frequency ratios and log reinforcermagnitude ratios. Each point was obtained by averaging choices and reinforcers from the last 65 trials of a block. All suchdata from both the frequency (open squares) and magnitude (filled circles) experiments are plotted. The planes throughthe data are fits of the generalized matching law to the entire data set for each monkey.


patterns. Instead, they show that the monkeystend to stay at the richer alternative, switchingbriefly to sample the leaner alternative. Baum,Schwendiman, and Bell (1999) observed thispattern of behavior in pigeons under concur-rent VI VI schedules, terming it ‘‘fix-and-sample.’’ The runlength analysis highlightsthe fact that the monkeys’ choices on in-dividual trials were not independent duringsteady-state behavior. These response-by-re-

sponse dependencies could have arisen fromthe effects of past reinforcers and/or pastchoices, but how this occurs cannot be revealedthrough sequential statistics like runlengthsbecause they do not take into account whenreinforcers were delivered. In the next sectionwe characterize these response-by-response dy-namics by constructing statistical models thatpredict choice on each trial based on the historyof reinforcers and choices.

Fig. 5. Mean runlength as a function of preference. Each point represents the mean runlength in the last 65 trials ofeach block, plotted separately for choices of the rich (circles) and lean alternatives (3s). The data are plotted on log-logcoordinates, and the points are jittered slightly along the abscissa to reveal points that otherwise would have stacked ontop of each other. The lines represent mean runlengths predicted by a Bernoulli process that allocates choiceindependently from trial-to-trial, as in a series of independent weighted coin flips. The left panels show examples fromsingle conditions for each monkey. The right panels show runlengths for all the data combined; for clarity, the datapoints for the conditions used in the left panels are not included in the pooled data.


Dynamic Analysis

We used response-by-response models topredict choice on each trial using the pasthistory of reinforcers and choices. Here weintroduce some necessary notation. Let cR,i

and cL,i represent the choice to left andright alternatives on the ith trial. Thesevariables are binary; a value of 1 indicatesa choice of a particular alternative. In orderto fit a model to the data, we seek to estimatethe probability, pR,i, of choosing the rightalternative (i.e., the probability cR,i 5 1). Weassume that past reinforcers and choices arelinearly combined to determine choice oneach trial, which allows us to write the logisticregression,

logpR ,i

pL,i

� �~Xj~1

aR ,j rR ,i{jzXj~1

aL,j rL,i{j

zXj~1

bR ,j cR ,i{jzXj~1

bL,j cL,i{jzc,

ð3Þ

where r is the magnitude of reinforcement inmilliliters of water received for choosinga particular alternative on the jth past trial,and zero otherwise. Like the generalizedmatching law, Equation 3 implies that theprobability and the magnitude of reinforcershave independent effects (i.e., they combinemultiplicatively). The a and b coefficientsmeasure the influence of past reinforcers andchoices, and the intercept term c capturespreference not accounted for by past reinforc-ers or choices, similar to the bias term in thegeneralized matching law. The coefficientsrepresent changes in the natural logarithm ofthe odds of choosing the right alternative (orequivalently left since pL,i 5 1 2 pR,i). Themodel is linear in the log odds, but nonlinearin the probability of choice, which can berecovered by exponentiating both sides ofEquation 3 and solving for pR,i.

We simplify the model by assuming that theeffects of past reinforcers and choices aresymmetric; a past reinforcer or a choice to theright alternative changes the odds of choosingright by the same amount that a past re-inforcer or a choice to the left alternativechanges the odds of choosing left. Note thatthis assumption is required for the past choices(but not past reinforcers) in order to be ableto fit the model because there are only twochoice alternatives, cL,i 5 1 2 cR,i. Then, the

model reduces to

logpR ,i

pL,i

� �~Xj~1

aj rR ,i{j{rL,i{j

� �

zXj~1

bj cR ,i{j{cL,i{j

� �zc:

ð4Þ

Here a unit reinforcer obtained j trials in thepast increases the log odds of choosing analternative by aj if the reinforcer was receivedfor choosing that alternative, otherwise itdecreases the log odds by aj. This appliessimilarly to the effects of past choices, wherea significant bj means that the current choicedepends on a choice made j trials ago (Cox,1970).

Note that excluding the terms associatedwith the b parameters (forcing all bj 5 0)yields a model that only depends on thehistory of obtained reinforcers, producingresults similar to those obtained by transferfunction estimation (c.f., Palya et al., 1996,2002; Sugrue et al., 2004). Alternatively,excluding the terms associated with thea parameters (forcing all aj 5 0) yieldsa regression that only depends on the historyof choices taken. Including both a and b allowsus to assess the independent effects reinforcerand choice history have on current choice.

The intercept term c shifts preferencetowards one of the alternatives irrespective ofreinforcement (it captures a bias not due toeither reinforcer frequency or reinforcermagnitude). It also is possible that themonkeys were biased towards the rich alterna-tive, independent of its spatial location. This isdistinct from the effect captured by c, becausea bias towards the rich alternative will switchrepeatedly within a session (as the location ofthe richer alternative switched), so long as theanimal can actually identify the richer alterna-tive (Davison & Baum, 2003). We tested forthis effect by adding a dummy variable that was+1 on trials when the right alternative was richand 21 when the left alternative was rich. Thisallows for a bias towards or away from the richalternative depending on the sign of the fittedcoefficient for the dummy variable. We as-signed the identity of the rich alternativeaccording to the first reinforcer the animalobtained in a block, and this remained fixeduntil the first reinforcer obtained in thefollowing block, at which point the sign of


the dummy variable switched. A rich bias isseparable from the effects of past reinforcersbecause the first sum in Equation 4 onlycontributes when reinforcers are delivered(recall that reinforcers are not delivered onevery trial), whereas a bias towards the richalternative can influence behavior even whenno reinforcers are delivered.

The complete model therefore had fourcomponents: a dependence on past reinforc-ers and choices, as well as two kinds of bias,one for a constant bias towards an alternativeand another for a bias towards the richalternative. We fit this model to the data usingthe method of maximum likelihood (e.g.,Fahrmeir & Tutz, 2001; McCullagh & Nelder,1989). The model was fit twice for eachmonkey, once for all of the data pooled acrossreinforcer frequency conditions (32,126 trialsfor Monkey H and 24,337 trials for Monkey B),and again for all of the data pooled acrossreinforcer magnitude conditions (35,701 trialsfor Monkey H and 22,823 trials for Monkey B).The intersession spacing was dealt with bypadding the design matrix with zeros so thatreinforcers and choices from different sessionswere not used to predict responses from othersessions.

The results obtained from fitting the modelare plotted in Figure 6. The coefficientsrepresent changes in the log odds of choosingan alternative due to a unit reinforcer (upperpanels) or a previous choice (middle panels).The coefficients are plotted for models fitusing a history of 15 trials. We address howlong a history is required by the mostparsimonious model in a later section; extend-ing the history does not affect the resultspresented here. The change due to a reinforc-er j trials in the past (upper panels) can bedetermined by reading off the coefficientvalue from the ordinate and multiplying it bythe magnitude of reinforcement in milliliters.For example, the effect of a typical 0.5 mlreinforcer four trials in the past for Monkey Hin the probability conditions is 0.83 3 0.50 50.42. This is an increase in log odds, soexponentiating gives an increased odds of1.52 for choosing the reinforced alternativeon the current trial. The pattern of results forreinforcer history is similar for both animals;the most recent reinforcers strongly increasedthe odds of choosing the reinforced alterna-tive again, and this influence decayed as

reinforcers receded in time. For both mon-keys, the coefficients for reinforcer historyare similar in shape for both the probabilityand magnitude conditions, although the coef-ficients are larger in the probability condi-tions. Relative to Monkey H, the coefficientsfor Monkey B are initially larger, but also decaymore quickly, indicating that recent reinforc-ers would shift preference more strongly, butthat their effects are more local in time.

The middle panels in Figure 6 indicate thatthe effects of recent choices are somewhatmore complicated. These coefficients shouldbe interpreted in the same way as thecoefficients for past reinforcers, except thatthere is no need to scale the coefficientsbecause the choice variables are binary. Thenegative coefficient for the last trial means thatchoosing a particular alternative decreased theodds of choosing that alternative again on thenext trial, independent of any effects due toreinforcer history. Alone, this coefficientwould produce a tendency to alternate. Posi-tive coefficients for the trials further in thepast than the last trial mean that choosinga particular alternative some time in the pastincreases the likelihood of choosing it again,independent of the effects due to past re-inforcers. Alone, these coefficients would pro-duce a tendency to persist on an alternative.

Taken together the model coefficients in-dicate that recent reinforcers and choices actto produce persistence on the rich alternativewith occasional sampling of the lean alterna-tive. This pattern comes about because largeror more frequent reinforcers increase thelikelihood of repeating choices to the alterna-tive that yields them. Momentary decreases inrich reinforcers eventually cause a switch tothe lean alternative, followed by a tendency toswitch immediately back to the rich alternativedue to longer-term choice history effects.

The bias terms are plotted in the bottompanels of Figure 6. There was little or nospatial bias towards either alternative, whichwas consistent with our steady-state analysisusing the generalized matching law. Therewas, however, a bias towards the rich alterna-tive that was larger for the magnitude condi-tions.

We now return to the issue of determiningthe length of reinforcer and choice historiessupported by the data. Equation 4 does notspecify a unique model; each combination of


lengths of reinforcer and choice historiesconstitutes a candidate model. In principle,one could include nearly all past reinforcersand choices in the model; however, this lacksparsimony, and the resulting parameters canbe highly variable because estimation is limitedby the amount of data available. Alternatively,including too few parameters leads to a poorfit. We used Akaike’s Information Criterion(AIC; Akaike, 1974) to select objectively a bestmodel given the data. The AIC estimates theinformation lost by approximating the trueprocess underlying the data by a particularmodel (see Burnham & Anderson, 1998). For

each candidate model, the AIC is computed as

AIC~{2ln Lð Þz2k, ð5Þ

where is the maximized log-likelihood of themodel fit and k is the number of parameters.Equation 5 enforces parsimony by balancingthe quality of each fit with the increase inmodel complexity brought by adding moreparameters; good fits obtained by adding moreparameters decrease the first term but alsoincrease the second term. This makes intuitivesense; at some point adding more parameters

Fig. 6. Coefficients for the fitted dynamic linear model as a function of the number of trials in the past relative to thecurrent trial. The coefficients for past reinforcers, past choices, and biases are plotted in the upper, middle, and lowerpanels, respectively. The vertical ticks in the upper panels represent the largest 95% confidence intervals for the pastreinforcers. Confidence intervals are not plotted for the other coefficients, as they were smaller than the symbols.


is less informative because the extra param-eters are fitting random variations in the data.Selecting the model with the smallest AICvalue identifies the model that is the bestapproximation within the model set to thetrue underlying process in an information-theoretic sense.

We computed AIC values for a family ofmodels in which we varied the lengths of thereinforcer and choice histories. We varied thelengths up to 50 trials each, and selected thebest model according to the minimal AICvalue across the 2,500 candidate models. Theresults are listed in Table 3. Note that onlyrelative differences between AIC values aremeaningful; the absolute values associatedwith any one model are not. The best modelsare very local, integrating relatively few pastreinforcers (average 14 trials) and choices(average 22 trials), which confirms the visualimpression provided by Figure 6. Part of thereason that the effects of past choice extendsfurther into the past than the effects of pastreinforcers is due to the fact that there aremany more choices than there are reinforcedchoices because subjects were not reinforcedon every response. This provides more datawith which to fit the choice history coefficients.This is why the confidence intervals for thechoice history coefficients are smaller thanthose for the reinforcer history coefficients inFigure 6.

For comparison, results for two simplermodels also are shown in Table 3; one includes

only a constant bias, which is similar to a modelthat simply guesses with a probability of 0.5 oneach trial, whereas the other adds the effects ofpast reinforcers (without past choices). Thedifferences in AIC values in Table 3 representthe degree of evidence in favor of the best-fitting model, and the simpler models wereselected to give a sense of the contribution ofeach model component. The larger thedifference the less plausible a model iscompared to the best model; values greaterthan 10 on this scale provide strong supportfor the model with the smallest AIC value(Burnham & Anderson, 1998). Even thoughincorporating the effects of past choices intothe model increases its complexity, its im-proved ability to fit the data makes up for thecost of adding these extra parameters.

The AIC measure indicates that the best-fitting model requires relatively few param-eters. However, this measure can only de-termine which model is the best of the set ofmodels considered. Even a set of poorly fittingmodels will have a ‘‘best’’ model, so wechecked the goodness-of-fit by asking how wellthe best model (a) predicts average choicebehavior including transitions between rein-forcer ratios; and (b) how well it captures fineresponse-by-response structure.

Figure 7 shows an example of the choiceprobabilities measured in a single sessionalong with the corresponding probabilitiespredicted by the best-fitting model. To conveythe fit of the model to average choicebehavior, Figure 8 shows choice probabilities

Table 3

Model comparison using Akaike’s Information Criterion. The three rows for each conditioncorrespond to three separate models, with each row representing a more complex model. Thethird row for each condition is the best-fitting model according to the AIC metric, and the DAICcolumn gives the difference between each model and the best model’s AIC value. The numbersin parentheses for the best model correspond to the number of past reinforcers and choices,respectively. The total number of parameters for the models including past reinforcers andchoices includes the two bias terms.

Monkey H Monkey B

Number ofparameters AIC DAIC

Number ofparameters AIC DAIC

Frequency conditionsBias 1 39451 12502 1 32062 11973+ Reinforcers 17 35901 8952 17 28297 8208+ Choices 43 (13, 28) 26949 0 42 (14, 26) 20089 0

Magnitude conditionsBias 1 46162 14829 1 30184 10757+ Reinforcers 17 41340 10007 17 25429 6002+ Choices 38 (16, 20) 31333 0 28 (11, 15) 19427 0


in a format like Figure 2. Also plotted arepredictions from the best-fitting model, whichwere averaged just like the actual choice data.The model predictions captured the steady-state average as well as the dynamics of thetransitions between blocks. There is a mispre-diction, however, that is obvious only in theextreme transitions for the magnitude condi-tions (lower right panel in Figure 8). For thesetransitions, the monkeys were quicker torespond to new reinforcer conditions thanthe model predicted.

Although Figure 8 reveals that the modelaccurately captures average behavior in ourtask, this comparison cannot determine wheth-er the model adequately characterizes finestructure at the response-by-response levelbecause it is an average over many transitions.The runlength analysis above (Figure 5)showed clear structure at the response-by-response level, and we tested whether themodel sufficiently characterized the fine struc-ture in behavior by analyzing model residuals,the response-by-response differences betweenthe predicted choice probabilities and theobserved choice data. These residuals repre-sent variation in the data not explained by themodel, and are useful because inadequatemodels will yield structured residuals (e.g.,Box & Jenkins, 1976).

We used autocorrelation functions to assesswhether the residuals showed any response-by-response structure. These functions representthe average correlation between residuals

separated by a certain number of trials. Arandom process that is independent from trial-to-trial, like a fair coin, has zero autocorrela-tion for any trial separation (a fair coin flippednow does not depend on the outcome of anyprevious flip). Significantly nonzero autocor-relations in the residuals, therefore, wouldreveal systematic failures of the model. Weestimated autocorrelations by shifting theresiduals one trial and computing the correla-tion between the shifted residuals and theoriginal, repeating this for each of 25 shifts.Figure 9 shows residual autocorrelationsfor the best-fitting model. The horizontalgrey lines represent approximate 95% confi-dence intervals for autocorrelations expectedfrom an independent random process.For both monkeys, the autocorrelationsstayed mostly within these confidence inter-vals, indicating that the residuals were largelyunstructured. For comparison, the thinnestlines in the bottom panels of Figure 9 arethe residual autocorrelations from a modelthat only incorporates reinforcer history (thechoice history regressors were excluded).This model is inadequate, and clearly indicatesthe need to incorporate the effects of pastchoices to account for the response-by-re-sponse structure in our task. Indeed, thestructure of the autocorrelation functionsfor this model reveals the underlying tendencyof the monkeys to alternate and persist asshown by the full model (middle panels ofFigure 6).

Fig. 7. Predicted and observed choice data for a single session of data from Monkey H (Condition 6). The dashedvertical lines mark the unsignalled block transitions. The data were smoothed with a nine-point moving average.


Taken together, the analyses suggest thata local linear model that includes an effect forpast choices accounted for behavior in ourtask.

DISCUSSION

We studied the choice behavior of monkeysin a task where reinforcement contingencieswere varied within a session. We showed thateven though the monkeys’ choice behavior wasvariable at a response-by-response level, behav-ior averaged over tens of trials was a lawfulfunction of reinforcer frequency and magni-

tude. Monkeys adjusted their choice behaviorin response to abrupt reversals in the prevail-ing reinforcer conditions, reaching an averagesteady-state of responding after obtainingrelatively few reinforcers under the newschedule conditions. Steady-state behavior atthe end of each block was well accounted forby the generalized matching law (Baum,1974). When we varied the relative frequenciesof reinforcement, we found that monkeyspreferred the rich alternative less than wouldbe expected if they strictly matched the ratio oftheir choices to the ratio of reinforcersobtained, a finding known as undermatching.

Fig. 8. Predicted and observed choice data aligned on the trial that a transition between ratios occurred. The upperpanels are data from conditions where reinforcer frequency was manipulated, and the lower panels are data fromconditions where reinforcer magnitudes were manipulated. The left panels show examples from single conditions foreach monkey; the right panels show averages across the entire data set. These data were compiled with respect to the twopossible posttransition ratios for each condition (see Figure 2 for details). The data were smoothed using a five-pointmoving average.


However, in the magnitude conditions wefound that both monkeys matched, or slightlyovermatched, their choice allocation to theratio of obtained reinforcers.

These results may appear at odds withfindings in other species, which typically showthat animals slightly undermatch for rate ofreinforcement and strongly undermatch formagnitude of reinforcement (e.g., Baum,1979; Davison & McCarthy, 1988). However,like us, Anderson et al. (2002) reported strongundermatching in monkeys performing understandard concurrent VI VI schedules for foodreinforcers. This was true despite the fact thattheir study, unlike ours, used a changeoverdelay. In addition, we may have observed

stronger undermatching because we reversedthe reinforcement contingencies many timesduring a single session. Indeed, strong under-matching is commonly observed under condi-tions like those we used, such as frequent(Davison & Baum, 2000) or continuous (Palya& Allan, 2003) schedule changes. Regardingthe high sensitivity to magnitude, the pub-lished data are varied, with observations ofnear-strict matching (Brownstein, 1971; Keller& Gollub, 1977; Neuringer, 1967) as well asundermatching (Schneider, 1973; Todorov,1973) being reported.

A principal feature of the choice task weused was the unpredictable and frequentchanges in reinforcer frequency and magni-

Fig. 9. Autocorrelation functions of model residuals. The thick black line and the dashed line are the autocorrelationfunctions of the residuals from the best-fitting models (see Table 3) for the probability and magnitude conditions,respectively. The top panels show examples from single conditions for each monkey; the bottom panels show averagesacross the entire data set. The thin black line in the lower panels is the autocorrelation function of the residuals froma model that does not account for the effects of past choices (only shown for magnitude conditions). The gray horizontalbands give approximate 95% confidence intervals for an independent stochastic process.


tude ratios within sessions. Periods of steady-state behavior were preceded by transitionsduring which the monkeys adjusted theirbehavior to new reinforcer conditions. Choicebehavior began to shift towards the richeralternative after relatively few reinforcers hadbeen delivered in the new reinforcer condi-tions, despite the fact that block changes wereunsignalled and unpredictable. Similarly rapidbehavioral adjustments have been observed inresponse to changes in the frequency (Davison& Baum, 2000; Dreyfus, 1991; Gallistel, Mark,King, & Latham, 2001; Mark & Gallistel, 1994;Sugrue et al., 2004) and magnitude of re-inforcement (Davison & Baum, 2003) underconcurrent VI VI schedules, as well as underother types of reinforcement schedules (Bailey& Mazur, 1990; Dorris & Glimcher, 2004;Grace et al., 1999; Mazur, 1992).

We found that choice behavior during bothsteady-state periods at the end of each block aswell as in the rapid shifts during transitionsbetween blocks were well characterized bya response-by-response model that linearlycombined past reinforcers and past choicesto predict current choice. We found a decayingeffect of past reinforcers with more recentlyobtained reinforcers more strongly influenc-ing choice behavior. Our analysis differs fromother methods that have been used to exam-ine local reinforcer effects in that the effects ofpast reinforcers were separated from theeffects of past choices (c.f., Davison & Baum,2000, 2003; Dorris & Glimcher, 2004; Mark &Gallistel, 1994; McCoy, Crowley, Haghighian,Dean, & Platt, 2003; Sugrue et al., 2004).Analyses that do not separate these effects canbe more difficult to interpret. Behavior that isindependent of reinforcement, such as a ten-dency to alternate between two actions, mayappear to be caused by reinforcement ifsequential patterns of choice are not analyzedas well. For example, estimating the likelihoodof repeating a choice to an alternative byaveraging responses or response ratios fora number of trials following each reinforcerto that alternative often produces oscillatoryconditional probabilities (e.g., Bailey & Mazur,1990; Davison & Baum, 2000; Mazur, 1992),which these authors recognize is due tosequential structure in choice behavior. Theseoscillations can obscure the structure of theeffects due to reinforcers. In our analysis,when the effects of past choices were ac-

counted for, we found that the effects of pastreinforcers were highly local, becoming essen-tially ineffective by the time they have moved 5to 10 trials into the past (Figure 6). In contrast,excluding the effects of past choices can resultin biased estimates of the effects of pastreinforcers, which results from this reducedmodel’s attempt to account for the effects ofpast choices using only responses that yieldedreinforcers. We have found that even ifalternation is accounted for by incorporatingthe effects of the last choice, the estimates forthe effects of past reinforcers can be biasedupwards. In our data, failure to account for theeffects of enough past choices resulted inreinforcer effects that remained significant upto 30 trials into the past. Separating the effectsof past reinforcers and choices may be ofneurobiological importance if the mechanismsof reinforcement learning are distinct fromthe mechanisms of structuring sequentialchoice behavior (e.g., Daw, Niv, & Dayan,2005).

The linear model we used is only onemethod to separate the effects of past re-inforcers from past choices. Another methodis to estimate conditional probabilities withrespect to time rather than responses in free-operant tasks, which effectively averages outchoice structure unrelated to reinforcement(e.g., Davison & Baum, 2002). Alternatively,one could treat specific sequences of choicesas units of analysis (Sugrue et al., 2004). Bothof these methods also reveal decaying effect ofreinforcers, but they are not focused oncharacterizing the effects of past choices.One advantage of including covariates for pastchoices in the manner we did is that it allowsone to assess quantitatively the relative effectsof past reinforcers and choices.

By accounting for the effects of past choiceswe showed that our monkeys have a short-termtendency to alternate as well as a longer-termtendency to persist on the more frequentlychosen alternative. Combined with the effectsof past reinforcers, this produced a fix-and-sample pattern of choice; there was a tendencyto persist on the rich alternative with briefvisits to the lean alternative. This sequentialpattern of choices was first explored underconcurrent VI VI schedules using detailedanalyses of runlengths (Baum et al., 1999)and interresponse intervals (Davison, 2004).Moreover, Baum et al. showed that the


sequential structure varied depending onpresence or absence of a changeover delay;pigeons adjust their patterns of choice behav-ior to respect the fluctuations in local proba-bility of reinforcement. Similar adjustmentshave been observed when animals have beentrained to exhibit random sequences ofbehavior (Lee, Conroy, McGreevy, & Barra-clough, 2004; Neuringer, 2002). These resultsshow that animals can adjust the degree towhich memories of past reinforcers andchoices influence behavior. Incorporatingchoice as a covariate in dynamic response-by-response models allows one to isolate thesechanges from those due to past reinforcers(Lee et al., 2004; Mookherjee & Sopher,1994).

Theoretical Models

The statistical model we presented is princi-pally descriptive. However, we can recast it ina different way for comparison with substantivetheoretical models. Doing so helps relate ourwork to neurophysiological work by makingclear which variables might be explicitlyrepresented by the nervous system. Figure 10shows our statistical model separated into twostages. In the first stage, the subjective value of

each alternative is estimated from past experi-ence, and in the second stage a decision isformed by comparing these estimates. It iscommon to split the decision process in thismanner (e.g., Davis, Staddon, Machado, &Palmer, 1993; Houston, Kacelnik, & McNa-mara, 1982; Luce, 1959). As we have sketchedit, the first stage includes the effects of pastreinforcers, which we think of as providingestimates of the rate of reinforcement. To seethis more clearly, we can write the logisticregression in terms of the probability of choicerather than the log odds,

p~1

1zexpP

a rR{rLð Þð Þ , ð6Þ

where we ignore past choices momentarily andsuppress trial subscripts for clarity. The termsS a(rR 2 rL) encapsulate our assumptionsabout the internal representations animalsform about recently obtained reinforcers.Because the model is linear in the parameters,the coefficients weighting past reinforcers canbe rescaled so they sum to 1. This equivalentformulation has a simple interpretation; pastreinforcers are weighted to provide localestimates of the difference in average reinforc-er rate from each alternative. This estimate has

Fig. 10. Two-stage description of choice. The first stage involves valuation of each alternative based on past reinforcerand choices, and possibly other factors like satiety. The second stage generates the choice by comparing the value of eachalternative and selecting one using a decision rule.


units of reinforcer magnitude per trial, and ispassed to the second stage to form a choiceprobability.

Many theoretical models of foraging andrepeated choice behavior assume that the rateof prey encounter or reinforcement is a funda-mental variable that animals use to achieve thecomputational goals of maximizing fitness orreinforcement (Stephens & Krebs, 1986). Rateestimation is usually implemented by a weight-ed average with exponential decay (Killeen,1981, 1994), which captures the intuitivenotion that because natural environments arenonstationary, recent events should weighmore heavily in memory than long-past events(Cowie, 1977). Alternative weightings havebeen proposed. For example, Devenport andDevenport (1994) propose a temporal weight-ing rule that weights past reinforcers hyper-bolically (with respect to units of time). InFigure 11, we replot the coefficients for pastreinforcers along with examples of exponen-tial and hyperbolic sequences of weights. Thevariability between subjects precludes deter-mining which particular form of weighting ismore appropriate for our data, but the generalshape of the reinforcer coefficients supportsthe idea that animals represent reinforcementrate using algorithms that approximate theselinear weightings.

How should we think about the effects ofpast choices? The decision stage of many

choice models uses rules including simplyselecting the most valuable alternative (Herrn-stein & Vaughan, 1980), matching choiceproportions to the local rates of reinforcement(Kacelnik, Krebs, & Ens, 1987), and thelogistic function we have used, which iscommonly referred to as the softmax (Sutton& Barto, 1998) or logit (Camerer & Ho, 1998)decision rule. None of the models using thesedecision rules explicitly allows for sequentialstructure in choice behavior beyond thatinduced by structure in the obtained reinforc-ers. This is true for models using an exponen-tially weighted average because all knowledgeabout past reinforcers is represented by a singlenumber; this formulation thus possesses anindependence-of-path (Bush & Mosteller,1955; Luce, 1959).

Models incorporating choice dependenceare motivated by the nature of the choice taskwe used. Because the probability of reinforce-ment grows geometrically with time or re-sponses spent away from the lean alternative intasks like the one we used, it eventually pays tochoose that alternative, but because its prob-ability is reset to its initial low value oncechosen it is never worth staying more than onetrial. Houston and McNamara (1981) provethat reinforcer rate is maximized by repeatinga deterministic sequence; an optimal subjectwould choose the rich alternative a fixednumber of times (dictated by the particularschedules) followed by a single choice of thelean alternative (see also Staddon et al., 1981).Thus, behaving optimally in tasks like oursrequires counting the number of choices sincethe last changeover. Empirical tests of thehypothesis that animals use trial counting tomaximize reinforcement rate are equivocal,with evidence for (Hinson & Staddon, 1983;Shimp, 1966; Silberberg et al., 1978) andagainst (Herrnstein, data published in deVilliers, 1977; Heyman, 1979; Nevin, 1969,1979) the hypothesis. This leaves the questionof optimality unresolved, but the evidence forsequential structure in choice behavior isinteresting because models that only estimatereinforcement rates are not designed togenerate these patterns. Machado (1993)remedied this by incorporating short-termmemory for past responses into his behavioralmodels. Appropriate use of recent memory forpast responses can generate tendencies to shiftaway from, or to persist on, recently chosen

Fig. 11. Comparison of coefficients for past reinforcerswith exponential and hyperbolic weightings predicted bytheoretical choice models. Each of the four curves hasbeen normalized to unit area. The time constant of theexponential is three trials, and the hyperbolic function isgiven by the reciprocal of the trial number.


alternatives (negative recency and positiverecency), and is necessary for producingpatterned sequences of responses. This cap-tures the idea that one can only remember somuch information, but it is distinct from whatone does with that memory, which presumablydepends on reinforcement contingencies andother factors (Glimcher, 2005; Neuringer,2002).

Neurophysiology

One of the most interesting uses of thebehavioral models described above is that theymay provide insight into the underlyingneurobiological mechanism of choice, thusallowing tests of the models at the mechanisticlevel. As a starting point, a literal interpreta-tion of models with the form in Figure 10makes two predictions. First, certain brainstructures may implement a decision rule,functioning to compare the subjective valueof each available alternative to generatechoice. Second, other brain structures mayimplement valuation mechanisms, functioningto encode the various components that definevalue in a particular environment. For ananimal foraging for food, evaluating whichpatch to forage at is a complex combination ofestimating the reinforcement rate at eachpatch, the likelihood that an alternative patchhas recently become richer, the risk of pre-dation, and so on. These variables must berepresented in the nervous system if they areto affect choice, and neuroscientists arebeginning to test these predictions by combin-ing behavioral experiments with techniquessuch as single-unit electrophysiology andfunctional magnetic resonance imaging(fMRI).

A number of laboratories have focused onthe role of the parietal cortex in the decisionprocess. Single neurons in this cortical regionhave been shown to encode the accumulationof sensory evidence indicating which alterna-tive will yield a reinforcer (Roitman & Shad-len, 2002; Shadlen & Newsome, 2001; Z. M.Williams, Elfar, Eskandar, Toth, & Assad,2003) as well as the probability and magnitudeof reinforcement (Musallam, Corneil, Greger,Scherberger, & Andersen, 2004; Platt &Glimcher, 1999). Recently, Dorris and Glim-cher (2004) and Sugrue et al. (2004) recordedfrom parietal neurons while monkeys wereengaged in choice tasks with variable rein-

forcement probabilities. They found that theactivity of neurons in parietal cortex wascorrelated with estimates of value derived byfitting models using reinforcement rate topredict the monkeys’ behavior (similar toFigure 10, but without taking into accountpast choices). These results indicate thatparietal cortex receives information aboutboth sensory and reinforcing aspects of theenvironment.

The covariation of neural activity in theparietal cortex with behaviorally relevant vari-ables likely reflects computations carried outin other brain areas. For example, the dopa-minergic neurons in the substantia nigra parscompacta (SNC) and the ventral tegmentalarea (VTA) appear to be important forestimating reinforcement rate. Experimentswith monkeys and humans have shown thatthese neurons encode a prediction error, thedifference between the reinforcer expectedand the reinforcer obtained within a trial(O’Doherty et al., 2004; Schultz, 1998). Bayerand Glimcher (2005) linked SNC activity toreinforcement-rate estimation using the expo-nentially weighted average described above.They reasoned that if the value of a choicealternative is computed using an exponentiallyweighted average,

vi~vi{1za ri{vi{1ð Þ|fflfflfflfflfflffl{zfflfflfflfflfflffl}prediction error

,

and SNC neurons encoded the predictionerrors for these computations, then theirresponses should be proportional to thedifference between the most recent reinforcer(ri) and an exponentially weighted average ofall past reinforcers (vi21). They showed thatthe number of action potentials discharged byindividual SNC neurons is proportional to themagnitude of the difference between thereinforcer expected (vi21) and the reinforceractually obtained (ri), thereby lending strongsupport to the hypothesis that dopamineneurons underlie the computation of pre-diction error. These neurons send denseprojections to the prefrontal cortex and thestriatum, and current evidence suggests thatthese areas may use a prediction error toestimate reinforcer rate (Barraclough et al.,2004; Haruno et al., 2004; O’Doherty et al.,2004).


Research is rapidly proceeding in a numberof brain areas, but one relatively unexploredaspect is how animals integrate reinforcementhistory with other aspects of their ongoingbehavior. We have shown that monkeys en-gaged in a choice task appear to integrate bothpast reinforcers and past choices. Incorporat-ing memory for both of these elements wasnecessary to account for the structured pat-terns of choice behavior we observed. This isimportant for guiding interpretation of thephysiological data; characterizing the effects ofpast reinforcers and choices allows a finerseparation of these effects at the neural level.Barraclough et al. (2004) present evidencefrom a repeated-choice task that some neuronsin prefrontal cortex encode the identity of thelast response, independent of whether the lastresponse was reinforced or not. This raises theintriguing possibility that frontal cortical areasare important for evaluating what choice tomake next in light of what choices were madein the past. This idea is consistent withevidence that frontal cortex and the basalganglia—which are densely interconnected—are important for the expression and learningof motor sequences (Hikosaka et al., 1999;Tanji, 2001; Tanji & Hoshi, 2001). Physiolog-ical recordings from these brain areas withinthe context of a choice task may reveal theirrole in the generation of temporally extendedpatterns of choice behavior.

REFERENCES

Akaike, H. (1974). A new look at the statistical modelidentification. IEEE Transaction on Automatic Control,19, 716–723.

Anderson, K. G., Velkey, A. J., & Woolverton, W. L. (2002).The generalized matching law as a predictor of choicebetween cocaine and food in rhesus monkeys.Psychopharmacology, 163, 319–326.

Bailey, J., & Mazur, J. (1990). Choice behavior intransition: Development of preference for the higherprobability of reinforcement. Journal of the ExperimentalAnalysis of Behavior, 53, 409–422.

Barraclough, D. J., Conroy, M. L., & Lee, D. (2004).Prefrontal cortex and decision making in a mixed-strategy game. Nature Neuroscience, 7, 404–410.

Baum, W. (1974). On two types of deviation from thematching law: Bias and undermatching. Journal of theExperimental Analysis of Behavior, 22, 231–242.

Baum, W. (1979). Matching, undermatching, and over-matching in studies of choice. Journal of the Experimen-tal Analysis of Behavior, 32, 269–281.

Baum, W., & Rachlin, H. (1969). Choice as time allocation.Journal of the Experimental Analysis of Behavior, 12,861–874.

Baum, W., Schwendiman, J., & Bell, K. (1999). Choice,contingency discrimination, and foraging theory.Journal of the Experimental Analysis of Behavior, 71,355–373.

Bayer, H., & Glimcher, P. (2005). Midbrain dopamineneurons encode a quantitative reward predictionerror signal. Neuron, 47, 129–141.

Box, G. E. P., & Jenkins, G. M. (1976). Time series analysis:Forecasting and control. (Rev. ed.). San Francisco:Holden-Day.

Brownstein, A. (1971). Concurrent schedules of response-independent reinforcement: Duration of a reinforcingstimulus. Journal of the Experimental Analysis of Behavior,15, 211–214.

Burnham, K. P., & Anderson, D. R. (1998). Model selectionand inference: A practical information-theoretic approach.New York: Springer.

Bush, R. R., & Mosteller, F. (1955). Stochastic models forlearning. New York: Wiley.

Camerer, C., & Ho, T. (1998). Experience-weightedattraction learning in coordination games: Probabilityrules, heterogeneity, and time-variation. Games andEconomic Behavior, 42, 305–326.

Cowie, R. (1977). Optimal foraging in great tits (parusmajor). Nature, 268, 137–139.

Cox, D. R. (1970). The analysis of binary data. London:Methuen.

Davis, D. G., Staddon, J. E., Machado, A., & Palmer, R. G.(1993). The process of recurrent choice. PsychologicalReview, 100, 320–341.

Davison, M. (2004). Interresponse times and the structureof choice. Behavioral Processes, 66, 173–187.

Davison, M., & Baum, W. M. (2000). Choice in a variableenvironment: Every reinforcer counts. Journal of theExperimental Analysis of Behavior, 74, 1–24.

Davison, M., & Baum, W. M. (2002). Choice in a variableenvironment: Effects of blackout duration and extinc-tion between components. Journal of the ExperimentalAnalysis of Behavior, 77, 65–89.

Davison, M., & Baum, W. M. (2003). Every reinforcercounts: Reinforcer magnitude and local preference.Journal of the Experimental Analysis of Behavior, 80,95–129.

Davison, M., & Hunter, I. (1979). Concurrent schedules:Undermatching and control by previous experimentalconditions. Journal of the Experimental Analysis ofBehavior, 32, 233–244.

Davison, M., & McCarthy, D. (1988). The matchinglaw: A research review. Hillsdale, NJ: Erlbaum.

Daw, N., Niv, Y., & Dayan, P. (2005). Actions, policies,values and the basal ganglia. In E. Bezard (Ed.), Recentbreakthroughs in basal ganglia research (pp. XX–XX).New York: Nova Science Publishers.

de Villiers, P. (1977). Choice in concurrent schedules anda quantitative formulation of the law of effect. In W.Honig & J. Staddon (Eds.), Handbook of operantbehavior (pp. 233–287). Englewood Cliffs, NJ: Pren-tice-Hall.

Devenport, L., & Devenport, J. (1994). Time-dependentaveraging of foraging information in least chipmunksand golden-mantled ground squirrels. Animal Behav-ior, 47, 787–802.

Dorris, M., & Glimcher, P. W. (2004). Activity in posteriorparietal cortex is correlated with the relative subjectivedesirability of action. Neuron, 44, 365–378.


Dreyfus, L. (1991). Local shifts in relative reinforcementrate and time allocation on concurrent schedules.Journal of Experimental Psychology: Animal Behavior Pro-cesses, 17, 486–502.

Fahrmeir, L., & Tutz, G. (2001). Multivariate statisticalmodelling based on generalized linear models (2nd ed.).New York: Springer.

Gallistel, C. R., Mark, T. A., King, A. P., & Latham, P. E.(2001). The rat approximates an ideal detector ofchanges in rates of reward: Implications for the law ofeffect. Journal of Experimental Psychology: Animal Behav-ior Processes, 27, 354–372.

Glimcher, P. W. (2002). Decisions, decisions, decisions:Choosing a biological science of choice. Neuron, 36,323–332.

Glimcher, P. W. (2005). Indeterminacy in brain andbehavior. Annual Review of Psychology, 56, 25–56.

Grace, R. C., Bragason, O., & McLean, A. P. (1999). Rapidacquisition of preference in concurrent chains.Journal of the Experimental Analysis of Behavior, 80,235–252.

Haruno, M., Kuroda, T., Doya, K., Toyama, K., Kimura, M.,Samejima, K., et al. (2004). A neural correlate ofreward-based behavioral learning in caudate nucleus:A functional magnetic resonance imaging study ofa stochastic decision task. Journal of Neuroscience, 24,1660–1665.

Herrnstein, R. J. (1961). Relative and absolute strength ofresponse as a function of frequency of reinforcement.Journal of the Experimental Analysis of Behavior, 4,267–272.

Herrnstein, R. J., & Vaughan, W. Jr. (1980). Meliorationand behavioral allocation. In J. Staddon (Ed.), Limitsto action: The allocation of individual behavior (pp. 143–176). New York: Academic Press.

Heyman, G. (1979). Markov model description of change-over probabilities on concurrent variable-intervalschedules. Journal of the Experimental Analysis ofBehavior, 31, 41–51.

Hikosaka, O., Nakahara, H., Rand, M. K., Sakai, K., Lu, X.,Nakamura, K., et al. (1999). Parallel neural networksfor learning sequential procedures. Trends in Neuros-ciences, 22, 464–471.

Hinson, J. M., & Staddon, J. E. (1983). Hill-climbing bypigeons. Journal of the Experimental Analysis of Behavior,39, 25–47.

Houston, A., Kacelnik, A., & McNamara, J. (1982). Somelearning rules for acquiring information. In D.McFarland (Ed.), Functional ontogeny (pp. 140–191).London: Pitman Books.

Houston, A., & McNamara, J. (1981). How to maximizereward rate on two variable-interval paradigms. Journalof the Experimental Analysis of Behavior, 35, 367–396.

Houston, A., & Sumida, B. (1987). Learning rules,matching and frequency dependence. Journal ofTheoretical Biology, 126, 289–308.

Hunter, I., & Davison, M. (1985). Determination ofa behavioral transfer function: White-noise analysisof session-to-session response-ratio dynamics on con-current VI VI schedules. Journal of the ExperimentalAnalysis of Behavior, 43, 43–59.

Iglauer, C., & Woods, J. (1974). Concurrent performances:Reinforcement by different doses of intravenouscocaine in rhesus monkeys. Journal of the ExperimentalAnalysis of Behavior, 22, 179–196.

Judge, S. J., Richmond, B. J., & Chu, F. C. (1980).Implantation of magnetic search coils for measure-ment of eye position: An improved method. VisionResearch, 20, 535–538.

Kacelnik, A., Krebs, J., & Ens, B. (1987). Foraging ina changing environment: An experiment with star-lings (sturnus vulgaris). In M. Commons, A. Kacelnik,& S. Shettleworth (Eds.), Quantitative analyses ofbehaviour, Vol. 6: Foraging (pp. 63–87). Mahwah, NJ:Erlbaum.

Keller, J. V., & Gollub, L. R. (1977). Duration and rate ofreinforcement as determinants of concurrent re-sponding. Journal of the Experimental Analysis ofBehavior, 28, 145–153.

Killeen, P. R. (1981). Averaging theory. In C. Bradshaw, E.Szabadi, & C. Lowe (Eds.), Quantification of steady stateoperant behavior (pp. 21–34). North Holland, Amster-dam: Elsevier.

Killeen, P. R. (1994). Mathematical principles of re-inforcement. Behavioral and Brain Sciences, 17,105–172.

Lee, D., Conroy, M., McGreevy, B., & Barraclough, D.(2004). Reinforcement learning and decision makingin monkeys during a competitive game. Cognitive BrainResearch, 22, 45–58.

Luce, R. D. (1959). Individual choice behavior; a theoreticalanalysis. New York: Wiley.

Machado, A. (1993). Learning variable and stereotypicalsequences of responses: Some data and a new model.Behavioural Processes, 30, 103–129.

Mark, T. A., & Gallistel, C. R. (1994). Kinetics of matching.Journal of Experimental Psychology: Animal Behavior Pro-cesses, 20, 79–95.

Mazur, J. E. (1992). Choice behavior in transition:Development of preference with ratio and intervalschedules. Journal of Experimental Psychology: AnimalBehavior Processes, 18, 364–378.

McCoy, A. N., Crowley, J. C., Haghighian, G., Dean, H. L.,& Platt, M. L. (2003). Saccade reward signals inposterior cingulate cortex. Neuron, 40, 1031–1040.

McCullagh, P., & Nelder, J. A. (1989). Generalized linearmodels (2nd ed.). London; New York: Chapman andHall.

McDowell, J. J., Bass, R., & Kessel, R. (1992). Applyinglinear systems analysis to dynamic behavior. Jour-nal of the Experimental Analysis of Behavior, 57, 377–391.

Montague, P. R., & Berns, G. S. (2002). Neural economicsand the biological substrates of valuation. Neuron, 36,265–284.

Mookherjee, D., & Sopher, B. (1994). Learning behaviorin an experimental matching pennies game. Gamesand Economic Behavior, 7, 62–91.

Musallam, S., Corneil, B. D., Greger, B., Scherberger, H., &Andersen, R. A. (2004, July 9). Cognitive controlsignals for neural prosthetics. Science, 305, 258–262.

Neuringer, A. (1967). Effects of reinforcement magnitudeon choice and rate of responding. Journal of theExperimental Analysis of Behavior, 10, 417–424.

Neuringer, A. (2002). Operant variability: Evidence,functions, and theory. Psychonomic Bulletin and Review,9, 672–705.

Nevin, J. (1969). Interval reinforcement of choice behaviorin discrete trials. Journal of the Experimental Analysis ofBehavior, 12, 875–885.


Nevin, J. (1979). Overall matching versus momentarymaximizing: Nevin (1969) revisited. Journal of Experi-mental Psychology: Animal Behavior Processes, 5, 300–306.

O’Doherty, J., Dayan, P., Schultz, J., Deichmann, R.,Friston, K., & Dolan, R. J. (2004, April 16). Dissociableroles of ventral and dorsal striatum in instrumentalconditioning. Science, 304, 452–454.

Palya, W. (1992). Dynamics in the fine structure ofschedule-controlled behavior. Journal of the Experimen-tal Analysis of Behavior, 57, 267–287.

Palya, W., & Allan, R. (2003). Dynamical concurrentschedules. Journal of the Experimental Analysis ofBehavior, 79, 1–20.

Palya, W., Walter, D., Kessel, R., & Lucke, R. (1996).Investigating behavioral dynamics with a fixed-timeextinction schedule and linear analysis. Journal of theExperimental Analysis of Behavior, 66, 391–409.

Palya, W., Walter, D., Kessel, R., & Lucke, R. (2002). Linearmodeling of steady-state behavioral dynamics. Journalof the Experimental Analysis of Behavior, 77, 3–27.

Platt, M. L., & Glimcher, P. W. (1997). Responses ofintraparietal neurons to saccadic targets and visualdistractors. Journal of Neurophysiology, 78, 1574–1589.

Platt, M. L., & Glimcher, P. W. (1999). Neural correlatesof decision variables in parietal cortex. Nature, 400,233–238.

Roitman, J. D., & Shadlen, M. N. (2002). Response ofneurons in the lateral intraparietal area duringa combined visual discrimination reaction time task.Journal of Neuroscience, 22, 9475–9489.

Schneider, J. (1973). Reinforcer effectiveness as a functionof reinforcer rate and magnitude: A comparison ofconcurrent performances. Journal of the ExperimentalAnalysis of Behavior, 20, 461–471.

Schultz, W. (1998). Predictive reward signal of dopamineneurons. Journal of Neurophysiology, 80, 1–27.

Schultz, W. (2004). Neural coding of basic reward terms ofanimal learning theory, game theory, microeconomicsand behavioural ecology. Current Opinion in Neurobiol-ogy, 14, 139–147.

Shadlen, M. N., & Newsome, W. T. (2001). Neural basis ofa perceptual decision in the parietal cortex (area LIP)of the rhesus monkey. Journal of Neurophysiology, 86,1916–1936.

Shimp, C. P. (1966). Probabilistically reinforced choicebehavior in pigeons. Journal of the Experimental Analysisof Behavior, 9, 443–455.

Silberberg, A., Hamilton, B., Ziriax, J. M., & Casey, J.(1978). The structure of choice. Journal of Experi-mental Psychology: Animal Behavior Processes, 4, 368–398.

Staddon, J. E., Hinson, J. M., & Kram, R. (1981). Optimalchoice. Journal of the Experimental Analysis of Behavior,35, 397–412.

Staddon, J. E., & Motheral, S. (1978). On matching andmaximizing in operant choice experiments. Psycholog-ical Review, 85, 436–444.

Stephens, D. W., & Krebs, J. R. (1986). Foraging theory.Princeton, NJ: Princeton University Press.

Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004,June 18). Matching behavior and the representationof value in the parietal cortex. Science, 304, 1782–1787.

Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning:An introduction. Cambridge, MA: MIT Press.

Tanji, J. (2001). Sequential organization of multiplemovements: Involvement of cortical motor areas.Annual Review of Neuroscience, 24, 631–651.

Tanji, J., & Hoshi, E. (2001). Behavioral planning in theprefrontal cortex. Current Opinion in Neurobiology, 11,164–170.

Todorov, J. (1973). Interaction of frequency and magni-tude of reinforcement on concurrent performances.Journal of the Experimental Analysis of Behavior, 19,451–458.

Williams, B. (1988). Reinforcement, choice, and responsestrength. In R. C. Atkinson, R. J. Herrnstein, G.Lindzey, & R. Luce (Eds.), Stevens’s handbook ofexperimental psychology. (2nd ed., pp. 167–244). NewYork: Wiley.

Williams, Z. M., Elfar, J. C., Eskandar, E. N., Toth, L. J., &Assad, J. A. (2003). Parietal activity and the perceiveddirection of ambiguous apparent motion. NatureNeuroscience, 6, 616–623.

Received September 29, 2004Final acceptance July 17, 2005


DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING ...

Documents

Transcript of DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING ...