SECURITY CLASSIFICATION OF THIS PAGE

REPORT DOCUMENTATION PAGE~PORT SECURITY CLASSIFICATION lb. RESTRICTIVE MARKINGS

fLn 2a. SECUR.TY CLASSIFICATIQN AUTHORITY 23 DISTRIBUTION/A VAILABILITY OF REPORT

S2b. DECLASSIFICATION/DOWI%(GRADING SCHEDULE

A. PERFORMOING ORGANIZA-ION REPORT NUMBERIS) 5 NIkRNG ORGANIZATION REPORT NUMBERIS)

N6~R-I a( d9 -1 851

6&. NAME OF PERFORMI NG OIGANIZATION 6b. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATION

S Depaxtiait of Psy410ooy (Iappicable) MINLI University of Massaahusetts ______

6cOADRESS (City. Slawe and ZIP Code) 7b. ADDRESS (City. State and ZIP CadetT inHall Building 410

Antierst, MA 0100i Boling AFB, DC 20332

Bo. NAME OF FUNDING/SPONSORING 18b. OFFICE SYMBOL. 9.PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER

ORGANIZATION J(It applicable) ADR-301Sc. ADDRESS (City. State and ZIP Code) 10. SOURCE OF FUNDING NOS.

Building 410 PROGRAM PROJECT TASK WVORK UNITBoln FDC232ELEMENT NO. NO. NO. NO.Boiin ABDC203261102F 2312 Al

11. TITLE (Include ziecigy, Clsicatiwaon) Biolo ical Investi-gations of Adaptive Networks -:m -oto of Conditioned Rasnnz

PERSONAL AUTHOR (S)

John W. IYbore,YPE OP REPORT 13b. TIME COVERED 14. DATE OF REPORT (Yr.. Mo.. Dowl 15. PAGE COUNT

rocc~tp~ozh'''FROM-5/1/84 TOU18 27 Nov 84 1416. SUPPLEMENTARY NOTATIO~N

17. COSATI CODES 18. SUBJECT TERMS lContinue on reverse it necessary and identify 67 611ch number)

FIELD GROUP SUB. GR.

19. ABSTRACT (Contirnue on reverse if noceusary and identify by bloc~k numb~r, .

S ELECTEJANO 5 199~DMT N1!T 6-TAThbiT

ApFrWVV I pvbb6 mk0czaI -s~budoa Uok~miod 90 01 04 1 2P20. DISTRIBUTIONiAVAILABILqTY OF ABSTRACT 21. ABSTRACT SECURITY CLASS4IATION

.ASSIFIED/UNLIMITED0 C' SAME AS RPT. 0 OTIC USERS0

22. NAME OF RESPONSIBLE INDIVIDUAL 22b. TELEPHONE NUMBER 23L OerICE SYMBOLj - i IIceude Arta CadetID FORM 1473, 83 APR EDITION OF I JAN 73 IS OBSOLETE.

SECURITY CLASSIFICATION OF THIS PAGE

.Zze. wp-

X e2- /670.?

DEPARTMENT OF PSYCHOLOGY November 1984

Dr. William 0. BerryProgram ManagerLife Sciences DirectorateAir Force Office of Scientific

Research (AFSC)Bolling Air Force Base, D.C. 20332

Re: AFOSR 83-0215(Adaptive Networks)

Dear Bill:

This Research Progress and Forecast Report covers the period fromMay 1, 1984 to the present. There have been a number of significantdevelopments in our research on adaptive neural networks using theclassically conditioned nictitating membrane (NM CR) in rabbit as a modelsystem.

As you know, we are placing a greater emphasis on the theoreticalaspects of learning than originally proposed. We seek to develop andevaluate real-time mathematical models that have potential applicabilityto learning in the Al domain. Our primary goal is to assess computa-tional versionsof these models against behavioral and physiological dataon the NM CR using simulation experiments. Our most recept efforts havefocused on three classes of models. N

1. Physiologically Constrained Sutton-Barto Model. This algorithmwas originally developed by Jonn E. Desmond. With additional work byNeil Berthier, and with advice from Rich Sutton, this model has beenrendered into a form that can make predictions about CR topography andthe firing pattern of neurons related to the CR. The original version ofthe model could do this reasonably well for the case of a single CS pairedwith the US in a forward-delay paradigm. The model has now been general-ized so that it can predict CR topography (or simply associative strength)and single-unit physiological data-within complex training paradigms thatinvolve two CSs with independent on-and off-times with respect to eachother and the US. __ These paradigms are analogous to spatial and temporalcredit-assignment pr'blems in reinforcement learning. Simulations are nowperformed on the University's VAXEN computer network, the same system usedby Andy Barto and hig collaborators in the Department of Computer andInformation Science'(COINS). A representative simulation of a two-CSconditioning experi gn is included with this report.

;r '' - lea

Dr. William 0. Berry 4November 27, 1984Page Two

2. Physiologically Constrained Moore-Stickney Model. The Moore-Stickney model is a real-time rendering of an attentional-associativelearning model developed originally by N. J. Mackintosh in England.This model has been applied to classical conditioninq in complex (multi-ple-CS) paradigms and to goal-seeking behavior. Its main success todate has been to describe the effects of hippocampal lesions in associa-tive learning tasks such as these. Nestor Schmajuk and I are completinga paper describing simulation experiments with the latest variant of theMoore-Stickney model. As part of his dissertation project, Schmajuk isapplying the model to the problem of CR topography and correlatedneuronal activity.

3. Real-Time Pearce-Hall Model. An alternate to the Mackintoshtype of attentional learning model was proposed by Pearce and Hall inEngland. Schmajuk and I have been developing real-time computationalvariants of the basic Pearce-Hall model for application to networklearning problems involving the hippocampus. The results of some simu-lation experiments under this model have been submitted for publicationand additional simulations of hippocampal lesion effects are describedin the paper mentioned in the preceding paragraph.

Theoretical activity has been intense this past few months becauseof the month-long visit by Dr. E. J. Kehoe of Australia. Kehoe and hiscollaborators have generated much of the behavioral data from the NM CRpreparation on multiple-CS effects. His visit in September set theoccasion for a series of seven seminars, Computational Learning Modelsand Test Beds with presentations by Barto, Desmond, Kehoe, Moore andSchmajuk with significant contributions from Neil Berthier andRich Sutton.

Since Kehoe's departure, we have met formally to consider predictionfrom several models regarding an experiment performed by my colleague,Joe Ayres, with relevance for real-time models. Ayre's experimentconcerned the effects of extending a CS in time beyond the reinforcingevent. Sutton, Desmond, and Schmajuk discussed the question from the view-point of several models, including: the basic Sutton-Barto Model, avariant of the Sutton-Barto Model designated by Sutton as the AdaptativeHeuristic Critique with Discount, Desmond's physiologically constrainedversion of the Sutton-Barto Model, and Moore-Stickney Model of attentional-associative networks. These formal meetings have been extremely valuablebecause they (a) enhance the groups' understanding of the various models,(b) familiarize colleagues and students with computational approaches toassociative learning, and (c) generate ideas for experiments that canprovide clear cut discriminations among models. In this regard, Kehoe andI plan to conduct some behavioral experiments relative to these models inparallel in our respective laboratories.

Most of our theoretical work has relied on the experimental literatureon behavioral aspects of the NM CR, i.e., on measures of associativestrength and on CR topography. Neuronal schemas that represent the various

Dr. William 0. BerryNovember 27, 1984Page Three

brain systems that might perform a given computational function within agiven model are based primarily on data from lesion studies. For example,in the Moore-Stickney Model the hippocampal formation is portrayed as asystem that reduced the salience or associability of stimuli depending onthe status of the associative network underlying behavior, an assumptionbased on observed effects of hippocampal lesions. The behavioral/biolog-ical foundations of such schemas will improve as relevant physiologicalevidence begins to surface. It is for this reason that we are accumulat-ing single-unit recordings concurrently with behavioral testing. Successin this domain demands that physiological recordings include neuronswithin brain structures known to be essential for the NM CR. (Our AnnualTechnical Report dated May 20, 1984 covers these points more fully). Inaddition, it is necessary to have computer-assisted statistical techniquesthat can quantify the relationship between neuronal firing and the fine-grain details of the behavioral CR. John E. Desmond has developed animpressive arsenal of quantitative tools for this purpose. Examples areincluded with this report. These same quantitative tools can be extendedto quantification of the relationship between theory-generated neuronalfiring and the activity of actual CR-related neurons.

There has been an important development concerning brain regionsessential for the NM CR. Contrary to published reports as recent as lastyear, it now appears that cerebellar cortex as well as deep nucleus inter-positus is essential for the NM CR. This came to light from lesion experi-ments by Yeo, Hardiman, and Glickstein at University College London, agroup with whom I have collaborated for several years. A lesion confinedto a portion of cerebellar cortex known to anatomists as the simplex lobe(a.k.a. hemispheric lobule VI, or HVI, for short) profoundly disrupts-apreviously acquired NM CR while having no effect on the underlying reflex(UR). In short, lesion of HVI, but no other portion of cerebellar cortex,produces the same deficit as does lesions of "downstream" premotor compo-nents of the cerebello-rubro-circuit essential for the CR.

I was able to examine relevant histological material and data duringa visit to the London laboratory this past June, and we have reproducedthe effect of HVI lesions in our laboratory. These lesion results, andother anatomical considerations, suggest that neurons within HVI arecrucial for the learning and execution of the NM CR. We have recentlybegun recording from single neurons of HVI during behavioral testing. ) ForWe are especially interested in the relationship between the Purkinjecell activity and the CR because Purkinje cells are featured prominently :in several mathematical models of cerebellar learning (e.g., Albus,Gilbert, Marr). Photocopies of the experimental set up for recording ,cedfrom Purkinje cells and sample data obtained by Neil Berthier areincluded with this report.

iDistribution/

- Availability Codes

, Dist s,"o@

Dr. William 0. BerryNovember 27, 1984Page Four

In order to establish the involvement of Purkinje cells in theNM CR, it is necessary to record from them during behavioral testing.There are a number of questions we intend to pursue:

1. Is there a relationship between Purkinje cell activity andCR topography?

2. What is the nature of this relationship and how well is itdescribed by computational models?

3. Are CR related Purkinje cells limited to HVI, or do they existin other regions of the cerebellar cortex?

4. Assuming that Purkinje cells are causally implicated in the NMCR, precisely how do they influence other premotor components of the CR,e.g., nucleus interpositus, red nucleus, and supratrigeminal reticularformation?

I can provide more detailed information regarding any aspect of thisreport immediately on request.

Cordially,

John W. Moore, Ph.D.Professor of Psychology(Neuroscience and Behavior)Associated Professor ofComputer'and Information Sciences

JWM/acl

Enclosures

F TI3L, EJ hotcocreAph o+ rabbit in- recordinci rh~tt iisani flna. ha i bacri proared ± (Jr brain ser r-ccordi flI s wi th a ho] I cwijpeic~tz-a1 tIhrouch w'-hi ch rEcIorcii nc electrouies are introcducud into the

br a i i

1: 1 1 E F s cnt a;, y CF RT t ra:c i n I 1C' ms/ i. Trs3i ngle L.:(nd i5. c_-i oni ~ t 'ri al s ciowi ng upper tr.ACcE-) NIN _1C7L1V_'1:V Dild

OWE w r' f.; C CDi) -Si r IQ : S R Uf 1 tq s, if OI1-ui t r 1- -im n i mci 71 Z.*-1flC3J ~ No -(- a.~- I n'OO r c r C..? ac n ir jn~ q aloul' 1.0,I) ins; bet our -~

in o f:hy- r c); i --tun i t i i-i (-',n i rca1 I2B sn ow i i -i i n cr i a s ed a t 1 -' 'i t ou~r~itvj-Ih n.ro L-i: to a tone C2-- . Anl 0 -4- Lknit i.n Hl. ai 1/ toC cn L:.

Note ciecr.la i n "7i r",nr approl.< 1 matel y 50.' as be-f-ore C.R. 1). : cL'fli L :- 3 , i- C. L'L' tx-n an ex Li n c i on t ri al - I th no ECR. E. in on-t. ni1 iif rn-an , .-: f'c: in an i ncrease i n act ivi tv Loncurreni: ~i~ai-hr CR-'

toa ;.Cii : CZ~ F . TrhIoL S a TIO 6?WIi t a~ -j.n E d Ltr i rO 1 c r1 i w t .r: tn ECUsJ- and flrv CSR.

F]U I F .." Repr-escentatj e coalf2~ttr qenerat,_yd e-.at1r'U's-..mrhi ;L. .- 4 nd avforaqo._.d NM acti vi tv. Th~~ao are' Oaaid _'in -- t -ar -ur-

the Op 1~.I j~~v1 o~m~dby r . F. S~ol omo, adh o au

n. ufL-nd D r cje-ncya c; onae -S .r i -: The r i qIt-hand bar rnenoi-F' ;: =.i k: c.;- t~ it ca 3jfC i z,-or n o-rir c--- n 4 or C- d t.-i al r*

rwoarj A H . iuinarv. ci I: CC'+ trials w ith foL;trn 1 CF5hOW1 1 j, ypi CAI On--Linit . 'Ihe clotts i ndi cat~e ast at . t, cr v

Q on. 1.1 -iI C C. knt sr inri reIE ati non t o r)r e- C S a c ti v tv. h e~ rl t-o aprovi c;ea .o menuk 0+ teats c o r b i n ofn i z-.4 s 1 q n 1._ ca., k: Z I ku1 l t

rviI ..' I1 n yp o t h E-:si s r ej e t ioan r-ecai o ns o or 2 -.- D . L; Thun0- *r -; in A.~ I:u~t on 15 CS- tri1als wi1th nou-ca

in 1-7ima I d'ri no 16 CS+ trials. 1). it-ae safne Unit '~ in C on 1.CS3- -riala

F~~ 1 JFf. er-i Sti mulUS-tine hi atoqram nind &varaccr-d W-1iresonc. Over-- CS+ tr. al : in iAni mF.l, u-2. P. 1 h se unit *:ka in ~Aover 12 C~.~ext incti on trialsa with no C~s. Farir ] s H 7an d L7 ar e s a rn r yroli ncle1--% -1:n os, ;lt understand ino Pane]s C and ).C.Ti me-correl oarzam based on th(. data in Fi q. C.D. T ecr&~rrbased rin the data in F io. 3C.

FT CRE~.CO].'mri 1: Second-order conditi oni nq usi ng ~mis C7sa and no US. CS1 oft aoL is contic uous_: with CS'2 onU. I n itijalV's: CS .2 = .9,~ CS1 =. (A) . Y as a i-unction of time for a t r iin which Cc3,a Are seri ally prosented. The fUnction is a rePS~l t of 50conditici-nai trials. Hash ma rks on the aibscissa reoresent, from I.F~tto ri put , GS1 I nset . Cc22 oniset . CS22 c'1--'set. (D~). V as a fLinction aftrials -for CSI and C132. Plotted over 50 trials. (C) . Per ist imulIus,time hi-itooarin of a3imrLtlatfid neuronal aCtiVi tV aCCLIML1l ated -)ver 50)CSl-C62 prlesentatioris. Hash marks on abscissa are the saime as in A.CO 1lm n '2: Serial compound co~nditioning using a- 710C ms CS1. 350 nisC1,32. -ind )i's US3. CSL1 anH CC?2 coterminate, iind their o+ffset iscontAuk~uu w.i th LIS on~;C-t. Ini t ial V := t) for bath (,Sn ( D) , yaEs .4-furict 100 of ti ne i-or ci comouOLd StiIMULI S prosenta~ti on a* ter lt:)tri al s. H.osr marks on abscissa represont. from left to riaht. CSIonset. CS2 onset, US onset (and compound on+ffset). (E) . V as af unc t ion rof Li i Is f or- CS 1 an d CS2q pl1.ot ted over 100) t r ialIs. (F).

Per i c-11 i mui.I]us t ime hi stoqram W. si mu..-tr'ed neuronal act i vi tv-Cc . (7L.muTir ited T over I (l) oiIu(::..iii "_st i mcm u i ls r.resenI: z.i t 1 ,on-s. (Absci ssa h,.ashmir:.. ar idecntical to those? in D).

F [.,l:It'_-_ 'ii. L.iri ..n d inhi bi ticon, in ';hic:h two t:rial tvr)v'*are wreo --t . iFor L- I? i1 r ;t: t'ype.TZ * a rei nf or'c'ad -crImpc:h. Ct .-Cflh::)l 5 Z r 9 I ' : : 51 ]. L K.,a'lne#cut-> wr:.'"(:s:E;it.. ti,) .'-i Ut .-K.j .g'rllL .[: .i. thr- #L%: ~ ;:.to cio;13n:; wit We I' cinh!piuilG otat. Mie s38cC'fu tr: a; r vfD

oossEu on. v'1irunr' Ti c:,o:wr prc'snrtatio of CSt U-k Ioru. ISJ I ~id U,,oa bh q5" n5 in cO 'rat r 1US i ( 5 -'i. lS. -I t l orfl n . L i' V

'-'re ,, ,_- ",' ;Cf , GS a. (A). Y as a lIFc'ion o f time t-or b: n rri.,

!V':;. Wo mI rn ks C "ith? .i: a::sia repr-:-ers't. tr m l t: E. i t-cIxt-,t and me ... .t t either G-S+ or CZ-. (0). V a o func i i wtri i - n I. ..S--. plotted over WiO .triai.. I:. I:lu ; 1tim , -. at- o- mti.-ted neiuronal act vity to CS- .' ,s-tar . ,SOU : OHI cce a 'cc umuliatu oCcver Wn tr ial s stcrtinn a r or tnwir,it. . .t I' w, i I C-s trair n .no. ;b-ci.sua [ash m;ar- ar-e as 1 r col ; .Iin vi. . oris-nmuliis time M isto.:gir:m of- 'i3 mulat-d 'ro-:rc in

c'cr.:'. L- L o CS + presentationis. Spi tice couints are- accamul a ,.l -. se .r ibei in C.

4L

wL

jCD

A B

C D

E F

F--- -- - 2

FIGURE 2

A B

... • .U:... . "U .WL.~J

SUB: -.EN4 CR TRIALS21 CS=350 MS SUB: 72BT- NO CR TRIALS=15 CS=350 MS

CIIN=i0 MS V.CAL.=.2 CNTS, 8 VOLTS BIN=IO MS V.CAL.=.2 CNTS. 8 VOLTS

7 TEST WAS USED T TEST WAS USED

.C.5 Z-TAILED DF=-4 o5 -TAILED DF=74_-AD TTME-51) MS LEAD TIME=-770 MS

,I BINS OMITTED Q SINS OMITTED

CSC ONSET AT -% MAX AMP CR ONSET AT 5% MAX AMP

NEURAL ONSET AT SIG. SIN # 9 NEURAL ONSET AT SIG. BIN # I

.EAN SPIKES/TRIAL DEPICTED MEAN SPIkES/TPIAL DEPICTED

C D

-. LS=t CS=ScO MS SUP: ,7*.- TRIALS=C' CS=75o MS

? =:' .' € '-.L AL.=.4 --"IJTS, 10.6 6 VOLTS SIN=1,.' MS '..CAL.=.4 CNTSc : VOLTS

TEST :,AS USED C BINS OMITTED-Th,:,LED DF=-4 MEAN SPIRES/TRIAL DEPICTED

.-, ,F,= t*0 MS

Er:- .LMITTEDfr; ON'ET AT S% MAX AMP.'EURAL ONSET AT SIG. BIN # 5

]'E4&N SFII ES/TRIAL DEPICTED

FIGURE 3

r

Lfl

cr- -. -- LL

Lr/LEl -

L... /-- U ~ Z

Z c/

.- ~- L

-U---m

o LL

t,-. z

tF z

-j U.

L~C -NW

LO U.

Cr LrC', -1X.

1

J,tr U.

Z -

(NU,

'-I4,

/

I 4.I 4'

(N4,

4'4,

C'-

_ 4,

0 V. IL.

U.'

=

A4,

I..''A'

U'

C'.

4.

4,

S V V .~U-

4, 4'

4r4,a.

_____ _________ Ii4'

____ 4,

',' U

CC

.

-W

6

Al

'-4-

C'..CVVCD

VV

UNIVERSITY OF MASSACHUSETTSAMHERST

MEMORANDUM

FROM ........ . !...Q..4.. .................................................... DATE ........ S pte bZt1.,... 19. 4 ....................................

TO .. ......... t. ........................ ........................................................................................................

SU BJECT...C pu.tatL a.L..Le n g9..A deL .z.n d...T ,t4..B-ds ......................................................................

Jim Kehoe i visiting 4or% one month. He wLU be giving tJtee Zectures on hzAesea'tch Zito Compowzd S~tmdulu Conditionng. His res3earch provides data jot deve-Loping and evaZuating computational modeZ6 o6 conditioning, with speciaZ emphasiz onCR topography.

We ate chedu2tng a serie6 o6 tunch time Zectues beginnng Monday, September 10,1984. Other taWZ. on modetZng CR topogtaphy and/or teated topics ae alzo .6cheduted.

A tentative scedule appea6 betow: Time: NoonPet ce: Room 101, blddlZeex House

1. Monday, Septembea 10, 1984Jim Kehoe, "Se~&a Compound"

2. Wedneeday, SeptembeA 12, 1984Jim Kehoe, "Combination Rule"

3. Monday, September 17, 1984John Dezmond, "CR Topography and NeuophyoZogical Correlates"

4. Wedne-day, September 19, 1984"Intwoducton to AtteittonaL Theorile, with John Moore

5. Monday, Septembe/ 24, 1984Nestor Sdwiajuk, "Rea-Time AttertionaZ Modets and CR Topography"

6. Wednesday, SeptembeA 26, 1984Andy Baxto, "Maybe La yered Network "

7. Monday, October 1, 1984Jim Kehoe, "Goodbye and Lot o6 Lucie"

UNIVERSITY OF MASSACHUSETTSAMHERST

MEMORANDUM

FROM .............. John Moorq ..... ............ DATE. 20..Navw.ebex.. ........ ...................TO ............Ayre,.. Bat.=.. er.t der.,...Dea ond..ad..Sr-ba uk ...........................................

SUB JEC T ........... ......................................................................................................

1. Ayres presented the rationale and outcome of an experiment relevant toreal-time learning theories of interest to this group. A draft manu-script entitled, "Extending CS Beyond vs. Prior to Reinforcement" isattached.

2. Sutton reported that the data from Ayres' study are basically consistantwith the original Sutton-Barto model. However, the derivative model,Adaptive Heuristic Critic with Discount (AHC-D), predicts an asymmetricaloutcome such that extension of a CS beyond the US causes a greater reduc-tion of associative value than an equal extension in a forward direction.

3. Schmajuk presented the results of simulation experiments with a versionof the Moore-Stickne des-cribed in a draft-manuscript entitled, "Varia-tions of CS Effectiveness Revisited: Modified Attentional Models ofClassical Conditioning". This model, as does ACH-D, predicts the asym-metry whereby post-US extension of a CS causes greater reductions ofassociative value than does forward extension of the CS.

4. Desmond presented simulations from the physiologically constrainedversion of the Sutton-Barto model that he developed with Neil Berthier'shelp. These simulations also indicated asymmetries as noted above.

The group seemed to agree that:

1. The Ayres' experiment was indeed relevant to real-time models.

2. A simpler experiment with the "blocked" CS would likely yield differentanticipated results than those under consideration here.

3. Anticipated outcomes from the point of view of the various models dependcritically on parameters. Thus, for example, simulated experiments onCS extensions before and after a CS depend on the duration of the CS,knowledge of the optimal ISI, and/or rates of recruitment and decay oftheoretical variables. These considerations should guide choice ofdesigns and parameters of real experiments.