DOCUMENT RESUME ED 390 912 TM 024 257 …ED 390 912 AUTHOR TITLE INSTITUTION SPONS AGENCY PUB DATE...

ED 390 912

AUTHORTITLEINSTITUTION

SPONS AGENCY

PUB DATECONTRACTNOTEPUB TYPE

EDRS PRICEDESCRIPTORS

IDENTIFIERS

ABSTRACT

DOCUMENT RESUME

TM 024 257

Goldman, Susan R.; Biswas, GautamAssessing Digital Circuit Design. Final Report.Vanderbilt Univ., Nashville. Learning TechnologyCenter.Office of Naval Research, Arlington, VA. Cognitiveand Neural Sciences Div.Aug 95N00014-91-J-1780; NR4421571-0354p.

Reports Evaluative/Feasibility (142)

MF01/PC03 Plus Postage.Cognitive Processes; *College Students; *ComputerSystem Design; *Electric Circuits; Higher Education;*Planning; *Problem Solving; Protocol Analysis*Digital Circuits; *Expert Novice Paradigm;Experts

The focus of this project was on characterizing andassessing design problem solving in the area of digital circuitdesign. Think-aloud protocols and computer traces of subjectproblem-solving behavior were used to elucidate the cognitiveprocesses involved in designing combinational and complex sequentialcircuits by: (1) studying the differences between two experts andfive intermediates in solving combination circuit problems; (2)

'characterizing planning behavior and its impact on the quality ofsolutions of 20 subjects for combinational circuits; (3) validatingthe effectiveness of traces collected by a design tool by two ratersin assessing problem-solving behavior for combinational circuits; and(4) characterizing and assessing problem-solving behavior of 11subjects designing complex sequential circuits. The combinationalcircuit studies revealed local planning in problem solving but littleglobal planning, and clear differences between intermediate andexpert problem solving did not emerge. On the other hand, the complexsequential circuit design problems revealed significant differencesbetween expert, intermediate, and novice problem solvers. Problemsolving was successfully modeled by integrating problemdecomposition, transformation and iterative refinement, andanalogy/prototype models. (Contains 8 tables, 3 figures, and 20references.) (Author/SLD)

*

**********,

Reproductions supplied by EDRS are the best that can be madefrom the original ocument. *

-':;; -'.

C\

hthCr)

U S DEPARTMENT OF EDUCATI )N(MK e 01 E ducattonai Research a .,drOvement

E OUCpTIONAL RE SOURCES INFORMATIONCE NTE R RIC,

Ms document has been reOrOduCed ISreceived horn the person Or orpanaatronnhoinahno it

C. Minor changes have been made to improvereoroduClion goalely

Pc...1150f new Or ocv mons stated .n this clef O.

.ment do nO1 neCeSSarily represent offiCial

. OE RI postiron or POOCvref-.

LL-1 ,-

"zi-stri"4.t- t Assessing Digital Circuit Design

:.y.i.!;;;154!".'.?;'±'.'_'.:::. ite.!,,,Ig.,.....-.. Z.-":

n - -..'Susan R. Goldma

.-,...;,...,:" --. ...,,-.:::-...,....,". ft..,-...,:- ...-, ,.. . .. - r .. ,, , ,..

:-;.,-..3:-.-4,,t. -4..4.4,tt..:. xi:, a-%7,' '..4 'P.% , -/ .1 -- - ,-c : fti.,1- .., ...-11.1:...: . 4 ....::- ,-... P.' .- .. ,: . .

- - ,

:,42.4.70."-*

":.1;,:

Van erbilt Univers].. .

j%;;,,i 1.1

1;4.t.7 - .

.!S - ^!.;

._ :fi,'- ,,,...,:..,. . ..,,,%.7 '._

1.....-../.:::: - . -E :V.- -4.-..,....1'....;

.1.,... z.:. -

^

. . .. ..-:

; anTAts,'-nt,t 4; v:

i*tt:7

-...... . - .

,..s... . , c;,..":: -,,..w, '. ArA,.. .:'::- ---;,'-;.,.r.e.=.;;,-,...s,---'.,-

'..,1. -.,";-,4::3171-11.;'"q"..t7:-ii.F,V ':::".Z.,......,--:'2-?-1-:..:2-2."...'S,,i,;;;., ..::' --

-....i.:si.r.... '.',:":!..:.'? ''', .

. . .. : ..,,,,_ -.74... ...,.., . ,.. ,.,;.,

... ...,.. t r:-.-....-r...74;1. , . Final RePort:r.;:.-,

--:7-,.:, :t.',..('..1'....,.... V--11.: r.....,- ,- .!. --',S,'1- Aki.., ! ..;,7. ''..eit.1.0 r, alt.:?';. ,.:'S".'

'''....:Cle, y iii-F ryii, ,-4,i4ft.;7,4":::.;.:*:.+4,-:!, ::: .7.:i-. . ,;,-. ....-

.. ''''''..j.-2.. '...e''':'....': 1" ' '.

-h ":-'4.1.11.::. i':-...

- ..nu. .. I.

sh t, - .

:

:he". ,.-?-

,'

,

r,e

-

I

;;44,?

Reproduction in whole or part is permitted for any purpose of the United States Government.

August, 1995

- -

:

,

This research was sponsored by the Cognitive Science Program of the Office of Naval Research,

under Grant Number N00014-91+1780, Contract Authority Identification Number NR4421571--

Nc_ -03.

Approved for Public Release; Distribution Unlimited.

BEST COPY AVAILABLE

REPORT DOCUMENTATION PAGEForm ApprovedOMB No. 0704-0188

Public rebbrtirr: burden fOr this collection of Intormabor is estimated tc average 1 hour der response. including Inc time for reviewing instructions. searching eiosting aata sources.gathering and maintaining the data needed, and completing anc revievring the collection of infcrmation Sena comments regarding this burden estimate or any other aspect ot thiscollection ot informatiOn. including suggestions tor reducing this owner, to Washington Heaacluarters Services. :gee:orate for Information Operations and Rep torts, 1215 JefiersonDavis Highwa, Suite 1204. Arlington. JA 22202-4302. and tc the Office o. Management aria Buoget. Raperwor. Reouction Project (0704.01138). Washington. DC 20503

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED

8/16/95 Final Report_15/01/91 - 04130/95

4. TITLE AND SUBTITLE 5. FUNDING NUMBERS

Assessing Digital Circuit Design: Final Report G - N00014-91-J-1680

C - NR4421571---03

6. AUTHOR(S)

Susan R. Goldman and Gautam Biswas

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATIONREPORT NUMBER

Learning Technology Center 95-1

Vanderbilt UniversityBox 45, PeabodyNashville, TN 37203 .

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING / MONITORINGAGENCY REPORT NUMBER

Cognitive Science Program (1142CS)Office of Naval Research800 N. Quincy StreetArlington, VA 22217-5000

11. SUThLEMENTARY NOTES

12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE

Approved for public release; distribution unlimited.

13. ABSTRACT (Maximum 200 words)

Cognitive diagnosis of expertise relies on characterizing expertise in the domain of interest. The focus ofthisproject was on characterizing and assessing design problem solving in the area of digital circuit design. Acombination of think-aloud protocols and computer traces of subject problem-solving behavior was used toelucidate the cognitive processes involved in designing combinational and complex sequential circuits. The

project had four components: (i) study the differences between experts and intermediates in solving combi-national circuit problems, (ii) characterize planning behavior and its impact on the quality of solutions forcombinational circuits, (iii) validate the effectiveness of traces collected !)11 a design tool in assessing problem-solving behavior for combinational circuits, and (iv) characterize and assess problem-solving behavior forcomplex sequential circuits. The combinational circuit studies revealed local planning in problem solving butlittle global planning, mainly because of the routine nature of the design process. Unlike other domains re-ported in the literature, clear differences between intermediate and expert problem solv.'ng did not emerg:.f.On the other hand, the complex sequential circuit design problem revealed significant oifferences betweenexpert, intermediate, and novice problem solvers. Problem solving was successfullymod( led by integratingthree cognitive models of human design: (i) problem decomposition, (ii) transformation ar.d iterative refine-ment, and (iii) analogy/prototype models.

14. SUBJECT TERMS 15. NUMBER OF PAGES

design, assessment, problem solving35

16. PRICE CODE

17. SECURITY CLASSIFICATION 18. SECURITY CLASSIFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF ABSTRAC'OF REPORT OF THIS PAGE OF ABSTRACT

Unclassified Unclassified Unclassified

NSN 7540-01-280-5500 Standard POrrn 298 (Rev 2-89)PreSCrIberi by ANSI SIO 239-18298 102

Assessing Digital Circuit Design

Susan R. Goldman and Gautam Biswas

Vanderbilt University

Final Report

August, 1995

Reproduction in whole or part is permitted for any purpose of the United States Government.

This research was sponsored by the Cognitive Science Program of the Office of Naval Research,under Grant Number N00014-91+1780, Contract Authority Identification Number NR4421571-03.

Approved for Public Release; Distribution Unlimited.

4

Abstract

Cognitive diagnosis of expertise relies on characterizing expertise in thedomain of interest. The focus of this project was on characterizing andassessing design problem solving in the area of digital circuit design. Acombination of think-aloud protocols and computer traces of subjectproblem-solving behavior was used to elucidate the cognitive processesinvolved in designing combinational and complex sequential circuits. Theproject had four components: (i) study the :iifferences between experts andintermediates in solving combir ational circuit problems, (ii) characterizeplanning behavior and its impact on the quality of solutions forcombinational circuits, (iii) validate the effectiveness of traces collected by adesign tool in assessing problem-solving behavior for combinational circuits,and (iv) characterize and assess problem-solving behavior for complexsequential circuits. The combinational circuit studies revealed local planningin problem solving but little global planning, mainly because of the routinenature of the design process. Unlike other domains reported in the literature,clear differences between intermediate and expert problem solving did notemerge. On the other hand, the complex sequential circuit design problemrevealed significant differences between expert, intermediate, and noviceproblem solvers. Problem solving was successfully modeled by integratingthree cognitive models of human design: (i) problem decomposition, (ii)transformation and iterative refinement, and (iii) analogy / prototypemodels.

Circuit Design Assessment1

Assessing Digital Circuit DesignONR Grant N00014-91+1780

The focus of this project was on the assessment of design problemsolving in the area of digital circuit design. Think-aloud protocolmethodology was employed to elucidate the cognitive processes involved inexecuting the design of these kinds of complex circuits. However, due to thelabor-intensive nature of protocol analysis, we were interested in ascertainingthe degree to which computer-based design tools would provide sufficientinformation to accomplish classificatory assessment of design expertise.

Previous research on the study of design processes suggested thatalthough it is a highly constructive process, there are some general principlesand heuristics that apply (e.g., Garrod & Borns, 1991). For example, Goel andPiro lli (1989) studied design tasks in different domains: architecture,mechanical engineering, and instructional design and collected a total of 12think-aloud protocols. From this database, the protocols of three participants,one from each domain, were discussed. The three participants varied inexperience: ten years' design experience in designing industrial trainingmaterial, six years professional experience as an architect, and limited designexperience in mechanical engineering.

From the protocol analysis, Goel and Piro lli identified eight significantinvariant; in the problem spaces of the participants in the three differentdesign disciplines:

extensive problem structuring,extensive performance modeling,personalized or institutionalized evaluation functions and stoppingrules,a limited commitment mode control strategy,constraint propagation,the role of abstractions in the transformation of goals,the use of artificial symbol systems,solution decomposition into leaky modules).

These invariants are generally consistent with the results of other researchersand describe design behaviors that are consistent across different domains.

I A leaky module is a nonindependent module. A decision made inone leaky module could have consequences in several others.


Designing Simple Digital Circuits

We began our investigation of design in digital circuits by examiningthe solution process for simple digital circuits, i.e., combinational logiccircuits. Our primary purpose in this study was to characterize theperformance of students just beginning to study the design and circuit layoutpro( ,s (intermediates) and compare it to that of solvers who were advancedgraduate students or had experience working in the field (experts). Althoughthere exist standard procedures for designing combinational logic circuits,gaining proficiency in their design demands that the designer is aware oftrade-offs between different performance parameters. Typical circuitperformance parameters include minimizing the total number of gates used,the input-output delay, and total power consumption. As a result, a numberof alternative design solutions may be possible, each of which satisfies thefunctional requirements, but only a small number (one or more) may beconsidered suitable when taking into account all the different requirements.Furthermore, the complexity of most design problems requires that they bebroken down into subproblems, each of which can then be solvedindependently. However, the decomposition process is often difficult andimproper decompositions often create subproblems that interact thuscomplicating their solution process. In such situations, designers are oftenfound to produce incorrect or suboptimal solutions.

The foregoing characteristics of the domain served as an initial set onwhich we wished to contrast expert- and intermediate-level designers. Theresults of that work are described in James, Goldman, & Vandermolen (1994)and are only summarized here.

A total of sev'n participants, two experts and five intermediates,designed a 1-bit full subtractor and were given two criteria for a "good"design. The first was to minimize cost (i.e., minimize the number of gatesused), and the second was testability of the circuit. The standard, or textbook,procedure for solving this problem relies on a sequence of six components.They are:

understanding the problem to determine and list the input andoutput variables, and the nature of their relations, from a worddescription of the problem,completing the truth table to express and make explicit thecomplete relations between input and output variables,constructing Karnaugh maps (K-maps) to generate minimalBoolean expressions that express the relation between individualoutput variables and the input variables,expressing the results of the K-map form as Boolean expressions,implementing these Boolean expressions as combi.lationalcircuits, andevaluating the circuits generated for correctness and optimality.


Think-aloud verbal protocols, paper and pencil solutions that pal,' 'nantsgenerated when they solved the problem, and retrospective intervie,constituted the raw data.

Based on previous expert-novice contrastive research, we expected thatthe more expert participants would exhibit more planning behavior than theless expert participants (e.g., Larkin, 1980), have stronger self-monitoringskills (e.g., Chi, 1978; Chi, 1987; Chi, Glaser, & Rees, 1982; Larkin, 1983; Miyake& Norman, 1979; and Simon & Simon, 1978), and consider alternativesolutions in attempting to achieve an optimal solution (Akin, 1977). Thecombination of these should lead to the more expert problem solvers makingattempts to optimize their solution, and also engage in more verification andevaluation, thus producing "better" solutions overall. We expected that theless-skilled participants would exhibit little planning in the beginning of theirsolution process, attempt to apply the standard solution methodologies in abrute-force manner, and stop to decide how to proceed when they hit animpasse (e.g., Paige & Simon, 1966). Therefore, even if they generated correctsolutions, more likely than not, their solutions would be suboptimal.

James et al. (1994) differentiated between the type and location ofplanning in the problem solution. Global planning was defined as planningthat concerned not just the component step in which it occurred, butsituations in which the designer was considering the impact of a sequence ofcomponent steps on the solution process, or the impact of one componentstep on another. Local planning dealt only with the solution component inwhich it occurred.

James et al. found that the two experts generally proceeded withoutdifficulty, but they failed to correct errors in their solutions, and only one ofthem engaged in global planning prior to beginning the problem solution.The latter two results were unexpected given the previous characterizationsof expert performance (e.g., Chi et al., 1988; Larkin, 1980). However, as hasbeen reported in other domains, James et al. found that all of theintermediates were characterized by an overall strategy of directly attackingproblem understanding by attempting the first step in the solution algorithm.For all of the participants, evaluation largely consisted of verifying the laststep (as opposed to checking if the solution matched the functionalspecifications stated in the word problem) and this did not lead to theirdetecting errors and suboptimal solutions. In addition, only one of theintermediates engaged in any global planning and it occurred not at thebeginning of the solution but in the middle. Localized planning occurredfairly frequently, especially among the intermediates, and typica"- occurredin the middle of the solution process. In general, the intermedia Les in thisstudy did not formulate global plans, but did exhibit local planningthroughout their solutions.

The James et al. study provides some information about the processesof solvers at various levels of expertise in a previously unexplored domain.


However, the James et al. study appeared to show considerable variability inperformance and in the planning processes. The high variability and thesmall number of p'articipants makes generalizing the results difficult. Theresults of James et al. (1994) indicate that intermediates tend to implement thestandardized circuit design process mechanically, and do not reflect on theconsequences and results generated in one step on subsequent steps. This wasclear from their solution traces in that (a) they did not attempt to compare theresults at subsequent steps with information at previous steps for verificationpurposes, and (b) they failed to look for common terms in the two outputequations so that the number of gates could be reduced in theimplementation.

In the expert group, one of the experts formulated a detailed global planprior to beginning problem solviing, while the other expert did not. Thisfinding does not necessarily contradict the literature on expert behavior. Theexpert who failed to verbalize a global plan appeared to have been carryingout a standard set of solution procedures that may have been automated and,therefore, not accessible to verbalization. These data suggest the possibilitythat if the intermediates started thinking about the steps in the solution at thebeginning of the problem solving process, they might be more likely toproceed through the steps in the algorithm more smoothly and direct theirattention to verification and optimization issues they did not otherwiseconsider.

A follow-up study to the "subtractor" combinational circuit designstudy was conducted to pursue the planning issue. A second purpose was tovalidate the computer-based design tool that had been developedsimultaneously with conducting the think-alcud protocol study. Planningand its relationship to expertise in solving design problems were the purviewof a Master's Thesis conducted by Carolyn M. James. Data for the DesignAssistant validation study were generated in the course of collecting theJames Thesis data.

Design of the Planning and Validation Study

Twenty participants designed two combinational circuits using theDesign Assistant tool. Participants, undergraduate or first year graduateelectrical engineering students, were paid volunteers. The majority ofstudents were enrolled in a senior-level digital design course. They had allcompleted one introductory course in digital circuit design and had verylimited design experience. Each participant was asked to solve two problems.Problem A was the Subtractor problem used in our first verbal protocol study(James et al., 1994). Problem B was the Job Skill problem, developed for thepresent study to be comparable to Problem A in terms of the processesnecessary for designing the circuit.

Problem A required the participants to design a one-bit full subtractor,taking into account several design constraints (see Table 1). The participants


were provided with a written description of the desired function of the circuit(subtraction) rather than being explicitly told the desired input-outputbehavior. The problem is similar to an ex -Imple often used in class (the addercircuit), but different enough so that nom. of the participants should havebeen able to generate the specific solution for this problem from memory. Theparticipants each should have had enough background knowledge to call tomind the standard framework (textbook strategy) for solving this type ofproblem. The circuit from Problem A, the Subtractor problem, requires threeinputs and two outputs.

Problem B, the Job Skill problem, required the participants to solve aproblem in which the input-output relations are described in words, ratherthan a description of the function of the circuit as in the Subtractor problem(see Table 1). Both problems are combinational logic problems and thus, thesolution process for both follows the same standard algorithm (textbookstrategy). The circuit from the Job Skill problem required four inputs and twooutputs. Both problems were judged by the instructor of the design course tobe approximately equal in difficulty and complexity.

The participants solved both problems using the Design Assistant tool.They were asked to think aloud during the solution process (c.f., Ericsson &Smith, 1991). Sessions were videotaped, with the camera focused on thecomputer screen. Audible comments by the participants were thus correlatedto particular actions on the computer. The Design Assistant software created atrace of the actions taken by the participant but could not trace theparticipants' verbalizations.

During the first of two sessions, each participant worked on a practiceproblem to familiarize them with the software, and then solved their firstproblem. The second problem was administered during a separate session inwhich explicit planning instructions were given to each participant. Eachparticipant solved all of the problems independently without timeconstraints. The participants were allowed to use reference materials (e.g.,textbooks) which were provided if necessary. The average time for solvingeach problem was two hours. Each participant was alone during problemsolving, but an experimenter was within hearing distance to answer anyquestions or to prompt the participant to continue talking during silentperiods.

After each participant solved the second problem, they were debriefedand had an opportunity to comment on the study and to make suggestionsfor future improvements to the software they had been using during thestudy.

Planning was manipulated by instructions prior to solving the secondproblem. (Problems A and B served as "first" or "second" problems equallyoften across participants.) Before solving the second problem each participantwas asked to formulate an overall plan for solving the problem and wasgiven an example of an overall plan (see Table 2). The example plan

iO


illustrated that generating an overall plan meant laying out things that needto be thought about, how to accomplish them, and what contingencies needto be considered.

Validation of the Design Assistant Tool

The methodology for validating the Design Assistant tool involvedcomparing the characterization of problem solving obtained by examining thecomputer-generated solution traces with that derived from analysis of thevideotapes of the problem solving. Problem solution was characterized by

overall solution strategies used,whether the Tri...th table, K-map and Boolean components wereapplied to the output oriables sequentially (one by one) orconcomitantly (parallel),whether the outputs were implemented independent of oneanother or together as one unit, andcorrectness of the final solution.

Coding and AnalysesFour measures were derived from the flow diagrams for each problem

solved by each participant. (Example flow diagrams appear in Figures 1 and 2.)These were overall solution strategies, intermediate steps strategies,implementation strategies, and completeness.

Overall Solution Strategies. Three types of Overall Strategies weredistinguished based on the component sequence that was executed by theparticipants. The first strategy was the Textbook Strategy and is indicated by alinear progression through the Truth Table, K-maps, Boolean equations,implementation and evaluation. The second was the Textbook Strategywithout Boolean equations, in which no Boolean equations were written foreither output. The third was the Textbook Strategy without K-maps. In thisstrategy, either K-maps were not used at all or they were only partiallyattempted and not used properly.

Intermediate Step Strategies. The Intermediate Step Strategies of thesolution were coded as either Serial or Parallel. A solution with Serial steps isone in which the K-map, grouping, and Boolean expressions were solved forone output before the other (see the middle section of Figure 1). A solutionwith Parallel steps is one in which the two outputs were worked on inparallel: K-map for first; K-map for second; Grouping for first, etc. (see themiddle section of Figure 2).

Implementation Strategies. The Implementation Strategies were codedsimilarly to the Intermediate Step Strategies. In a Parallel Implementation


Strategy, both outputs (e.g., Result and B-out for the Subtractor) wereimplemented and then simulated. In a Serial Implementation Strategy, oneoutput was implemented before the other.

Completeness. Intermediate steps and implementation could becompleted for only one of the two inputs or for both. If only one wasimplemented, the solution was less complete than if both were implemented.

Two raters were used. One rater coded each participant's two problemsfrom the videotapes and transcripts made from them. The other raterindependently coded each participant's two problems from the computer-generated trace of the solution.

Results of the AnalysesOverall Solution Strategies. There were two disagreements between

raters on the 20 problem A traces coded and two on the 18 problem B tracesthat could be coded. (Two problem B computer traces were lost due to asoftware malfunction.) The distribution based on the co,1ing done from thevideotape is provided in Table 3. This distribution shows that the majority ofsolutions followed the standard textbook algorithm. The next most frequentapproach was to leave out the Boolean component and work from the K-mapgrouping to an implementation. The disagreements between raters concernedwhether or not K-map or Boolean components had been included. For thefour disagreements, the computer trace indicated that these components hadbeen attempted, whereas the videotapes showed that the participants had notactually worked on the K-map or Boolean components. Disagreements of thistype were due to the fact that a participant could have opened a Booleanwindow but not actually worked on a Boolean expression. The computer traceindicated that the Boolean window had been opened but not what had beendone in that window, if anything. The computer-trace rater assumed that ifthe window had been opened work on K-maps or Boolean expressions hadbeen done. In contrast, the video showed whether or not any work had beendone in these windows.

Intermediate Steps Strategies. There were three disagreements onProblem A and two on problem B. The videotape coding showed that 12 ofthe problem A and 15 of the problem B solutions used a parallel strategy andthe remainder were serial. Disagreements between coders were notsystematic, i.e., one rater did not favor parallel versus serial designations. Thedifferences between raters were related to the fact that the software did notrequire a new window for the two outputs. It was not possible to tell from thecomputer trace whether the student was working on one or two outputs inone window If they were working on two, it was not possible to tell from thetrace if they were finishing the work on one before working on the other or ifthey were being worked on simultaneously. The videotape records clearly


indicate these features of problem solving. Rather, the rater working onlyfrom the computer trace had to make inferences about what had been workedon in the K-map window based on subsequent information in the trace.These software features led to the discrepancies between the raters.

Implementation Strategies. Here again, lack of sufficient detail in theinformation gathered in the computer traces resulted in the raters agreeingon only 9 of the problem A implementations and on only 8 of 18 problem Bimplementations.2 Many of the participants worked on the implementationsfor both outputs in the same window and the computer trace did not indicatethis. Based on the videotape-derived strategies, 12 of the 20 problem A and 15of the 20 problem B implementations were parallel. From the computer tracesonly 3 problem A and 4 (of 18) problem B implementations appeared to beparallel. The lower numbers can be attributed to the fact that this computertrace rater had no information on how the connections among componentswere being established. Thus, there was a lower estimate of parallelimplementation.

Completed Outputs. Just as lack of sufficient detail in the computertraces led to low rater agreement on the implemeniation strategies, there waslow agreement on the number of outputs implemented. Raters agreed on 9problem A and 8 problem B solutions. Based on the videotape-derivedinformation, only three problem A and two problem B solutions failed todeal with both outputs.

hi addition to the problems with the lack of detail in the DesignAssistant tool traces, information about the purpose for various actions wasnot captured. However, the Design Assistant tool was adequate for purposesof describing the overall strategy and intermediate steps strategies. The resultsof the study of planning (James, 1995) indicated that this information, whichis reliably derivable from the computer trace generated by the DesignAssistant, may be useful in distinguishing among levels of expertise.Effects of Planning on Digital Circuit Design

To further investigate the effects of planning during simple digitalcircuit design, the next study (James, 1995) examined the effects of anticipatoryplanning on the problem-solving processes involved in solving the circuitdesign problems. Twenty intermediate-skilled electrical engineering studentswere asked to think aloud while solving two design problems. Before solvingthe second problem, participants were asked to create a detailed global plan oftheir problem solving. Subsequently, these plans were categorized by anexpert into high, medium, or low expertise categories. The contents of theplans were exhaustively coded into three categories of statements. Theproblem-solving protocols were examined for different types of strategies and

2Details of the implementation steps, such as the connections betweengates, were not recorded.

1 3


solution quality.The main hypotheses of this study had to do with the effects of asking

participants to plan their solutions prior to actually beginning the design task.It was expected that planning would lead to increased quality of solution, asindexed by optimization attempts, number of component steps attempted,and correctness of the final solution. The.planning manipulation did notsignificantly affect any of the measures of solution quality. The reason for thelack of effects due to planning may have been because the plans were notperceived as helpful. Accordingly, a second set of analyses examined whetherparticipants' perceived helpfulness of the planning was significantly related tothe same three measures of solution quality as well as to solution strategy.However, there were no significant effects related to perceived helpfulness ofthe plan.

Although there were no significant effects of the planningmanipulation, there were significant relationships between the expert's ratingof the plans and (a) the contents of the plans, and (b) the overall problem-solving strategy. A content analysis of the plans indicated that participantswith more Planning-Related Statements in their plans were more likely toreceive higher plan ratings. Their plans included more action statements,explanations, and evaluations than did the plans that received medium andlow ratings from the expert.

One of the interesting findings of the study concerns the overallproblem solving strategies used by the participants who attained higher planratings. Participants who attained higher plan ratings employed Booleanequations in their problem solutions more frequently than the otherstudents. Given the background and capabilities of the participants, it seemsreasonable to conclude that the few students who understood the nature ofBoolean expressions and were willing to manipulate them directly (forpurposes of transformation and reduction) had a more completeunderstanding of the domain, especially the relation between logic gates andBoolean expressions. Most of the other students appeared to apply thecomponent-step procedures mechanically without a true understanding ofwhat they were achieving. This observation is further supported by thenumber of errors students made in their final solutions, and their lack ofability to detect and correct these errors. Both tendencies were related to expertplan rating.

Contrary to general expectations, providing an opportunity by askingfor an anticipatory plan did not increase the quality of the participants'solutions. Anticipatory planning is hypothesized to be most helpful insituations where the participant can reduce the problems to well-structured,familiar and simple forms or a set of simpler subcomponents (Scholnick &Friedman, 1987). The problems in the present study were intended to be well-structured. However, approximately half the errors made in solving theseproblems (48%) occurred in generating the truth table. An additional 39%

14


occurred in the K-map component. These errors resulted in participantstrying to solve rather arbitrary, unfamiliar, and ccmplex problems. On suchproblems, anticipatory planning would not be ap?ropriate nor expected tohave beneficial effects on problem solving (Scholnick & Friedman, 1987). Theoccurrence of errors in the early components rnay also explain why planrating was not predictive of correctness of the solution nor of minimizationand optimization measures.

The findings of the planning study si,ggest potentially fruitful areas offuture research, including investigations of the use of Boolean equations bymore expert designers. It is possible that the effective use of Booleanequations is a benchmark in the acquisition of expertise in this domain.Results from this study indicated that the global plans generated byintermediate participants were not detailed enough to describe and capturethe entire solution space. It might also be interesting to determine at whatpoint in the acquisition of expertise the student of design can and doesformulate a detailed global problem-solving plan spontaneously, and at whatpoint such plans can be formulated when explicitly prompted. Ourexperiences suggest that providing an accurate truth table may well facilitatestudying the relation between planning and problem solving.

Differences in overall and intermediate steps solution strategies weresignificantly related to the expert's plan ratings. These are two indices thatcould be reliably determined from the computer trace generated by the DesignAssistant tool. Thus it seems that efforts to develop computer-based,performance-oriented design process assessments is a fruitful direction.

Designing Complex Circuits

One of the difficulties encountered in the design process analysis of thecombinational logic circuit problems was their "beginning" nature. This focuswas necessary to pursue the expert - intermediate contrastive comparison wewere interested in. Yet, the basic nature of these circuits contributed todifficulties encountered by the experts. In practice, circuit design at this levelhas become automated. Combinational logic circuits are often elements ofmore complex systems that experts design. They are "off the shelf"components that can bel7lugged into larger designs without being generatedanew each time they are needed. Thus, experts rarely design these kinds ofcircuits once they have mastered basic and intermediate levels of circuitdesign. Accordingly, the other major strand of work in this project examinedthe design process for very complex digital circuits.

The complexity in engineering design can be attributed to the fact thatmapping from a specified functionality to a realizable physical structure (thedesigned artifact). Complexity arises from

the specified functionality. Complex functions often have to be broken downinto sub-functionalities that interact (cf. leaky modules (Goel & Piro Ili, 1991)).

Li


Composing the individual sub-functionality implementations becomes adifficult task.mediating the relation between function, structure, and behavior. Behavioracts as the intermediary between function and structure. A designerhypothesizes a structure, determines its behavior. and then compares this tothe functionality (expected behavior). This process is non trivial; it defines anon-linear iterative framework for the design process.additional criteria, such as performance parameters and aesthetics that may beimposed on the design task.

A primary goal of this part of the project was to identify the differencesin the processes employed by experts and novices in dealing N., rith complexdesign problems with the above characteristics. As discussed E arlier, theproblem with combinational circuit design was its routine anc: algorithmicnature. Furthermore, the availability of automated Computer Assisted Design(CAD) tools in this domain, implies that experts and practitioners are rarelyinvolved in the details of combinational circuit design. On the other hand,current digital design activity primarily focuses on systems, such asprocessors, computer subsystems, memory subsystems, and networkcontrollers that include elements of combinational and sequential circuitdesign. The complexity of their functional descriptions requiresdecomposition into subproblems to achieve solutions. In addition, theirbehavior descriptions are dynamic, i.e., their behavior at a certain time step isdependent on the state of the system at the previous time step, makinganalysis and understanding of such systems a more difficult task. Ourexperience with the simpler combinational circuit suggested that shifting theemphasis to complex design tasks would

allow the study of systematic strategies in generating designsolutions, andmake explicit the complex and multi-dimensional nature of expertproblem solving processes.

In studying complex circuit design activity, it became clear that good designershad to catch flaws and resolve discrepancies early in the design process toavoid being overwhelmed by the magnitude and complexity of the problemin later stages. Hence, we were interested in characterizing the processesinvolved in producing the final design, as well as the completeness andcorrectness of the final design itself.

Characterizing Complex Design Activity

Complex design tasks ha ve been classified into three broad levels of design


activity (Tong and Sriram, 1992):3

Design activities at the function level study the specifications of asystem and generate functional units that collectively meet therequired specifications.The transition level includes design activities that look tofunctional units and decide how to implement them as artifacts(e.g., gates and higher-level blocks such as shift registers andcounters). Actions at this level may also guide the coordination ofcomponent artifacts so that higher-level functionality is satisfied.Implementation level activities deal with the actual generation ofthe artifact. In complex design problems this often happens instages; for example, simulation, testing, and evaluation of partialdesigns.

Researchers have also described orthogonal gradients that characterizethe cognitive processes of the designer, independent of the artifact beingdesigned (Korpi, Greeno & Jackson, 1991, Goel & Piro lli, 1991, Ullman et al.,1988). In this work, we have found two descriptive levels of reasoningsufficient: the strategy and unit operator levels (for details see, Biswas et al.,1994). In keeping with our interests in studying planning behavior in designproblem solving, our characterization of design activity was focused on thestrategy level. A survey of past and present design research indicated a set ofhigh-level problem solving strategies for design.

Decomposition strategies subdivide a complex design task intosmaller sub-tasks until they are directly solvable. Examples of designsystems that employ decomposition are Hi-Rise (Maher, 1988) andR1 (McDermott, 1982).Transformation/Refinement strategies convert initial specificationsinto a final design solution through a sequence of primitiveoperator applications in some formal representation scheme (e.g.,shape grammars (Mitchell and Stiney, 1978)).Case-based strategies assume that a catalog of previous designsolutions is stored in memory; a new design problem prompts thedesigner to search through this catalog, and select one or moredesigns that best match the characteristics of the goal design.Examples of case-based approaches to design include the Bogartsystem (Mostow, Barley, and Weinrich, 1989) and Struple (Zhao andMaher, 1988).

We hypothesized the presence of each of these strategies in designers. The

3This is a broader and richer description of design activity than the overall,intermediate, and implementation strategies described for the combinational circuitproblems.

1 'i


possibility of multiple strategies in a single design solution provided a muchricher framework for characterizing the reasoning methods of designers incomplex problem solving.

Design can be looked at as an application of reasoning processes thatstart at the function level, go through a transition level, and produce animplementation-level description of the artifact. Several strategies can beapplied in this process. The directedness with which these strategies wereemployed and coordinated by our participants was used as the basis forassessing a designer's expertise relative to a design problem or class ofproblems.

A Study of Complex Circuit Design

Our objective in this study was to correlate planning behavior and theuse of strategies with levels of expertise. We selected a complex problem,designing a network controller to coordinate communications betweencomputers that are linked by cable. A more complete specification for thisproblem is given in Table 4. Our experience with this and similar problemssuggested numerous activities that would be observed across a range ofdesigners.

We administered the network controller problem to elevenparticipants. Three were considered experts: S8, a faculty member atVanderbilt University who teaches digital design and had designed digitalsystems for industrial applications, S7, who works in the communicationstechnology industry, and S11, a graduate student with extensive CMOS digitalcircuit design experience in industry. Two participants, S4 and S10, hadrelated industrial experience, but neither could be considered to be experts incomplex circuit design. Four participants (S2, S5, S6, and S9) were graduatestudents with little or no professional design experience. The other twoparticipants were seniors in the undergraduate program at Vanderbiltuniversity. They both had completed a senior level course in digital circuitdesign. However, participant S3 was regarded as a high-performing student,whereas S1 was considered to be average in academic performance.

Protocol AnalysisThe participants were given a hard copy of the problem description (see

Table 4), and asked to develop their solution with pencil and paper. Theywere requested to think aloud as they developed their solution, and the entiresession was videotaped. The participants could use any material (notes, books,etc.) that they felt would assist them in generating a solution. A completetranscript of each session was generated.

Later a group of raters (Biswas, Glewwe, Bhuva) studied the tapes,transcripts, and the participants' written output. To encode strategic activity,the following two-step process was used.


In Step 1, we encoded the implementation-level blocks created bythe participants, and the sequence in which they added, deleted, andmodified blocks as they went through the design process. This tracerecorded the designer's activities at the implementation level. Mostimplementation level blocks originated from a function-leveldescription and retained the same name as the functional block(e.g., serial-to-parallel buffer). Participants usually put down thisname when they drew this block as part of the implementation onpaper, or they verbally expressed the name of the block as they drewit on paper.In Step 2, we used the Step 1 trace and information from the tapesand transcripts to answer the set of questions that define ourframework for characterizing design activity (see Biswas et al., 1994for details). This provided information on strategic level behaviorof participants at the function, transition, and implementationlevels.

The trace generated in Step 1, in addition to providing information forcreating Step 2, also helped us check the correctness of the design. If theparticipant had errors in the design, the trace helped us understand whereand how a participant went wrong. The analysis was performed in two steps:(i) use the Step 1 and Step 2 traces to characterize and analyze the participants'solution processes, i.e., process characteristics, and (ii) use the Step 1 results toanalyze and evaluate the correctness of the final solution generated by theparticipants, i.e., product characteristics. Process characteristics were studied atthe function, transition, and implementation levels.

Results of the analysesProcess Characteristics. Tables 5, 6, and 7 summarize the individual

behaviors of our 11 participants along the function, transition, andimplementation levels, respectively.

Table 5 shows the evaluation of each participant at the functior level.For this analysis, we assumed a normative solution, consisting of 7 functionalblocks, and an intermediate form (IF) given by the flowchart in Fig. 3. Thisconstituted the 'gold standard' against which we compared participant s'solutions. We were interested in assessing how close to this solution eachparticipant came. Because participants might differ in the number andorganization of ftmctional blocks, we developed an algorithm for countingthe amount of 'functional' overlap. If a designer defined a functional blockthat covered more than one of the 7 specified blocks, all 'covered' blocks inthe normative solution were counted as considered. If the designer usedfiner-grain blocks than the ones specified, a set of finer-grain blocks werecounted asa larger normative block if the designer effectively 'implemented' this block


with appropriate links between the fine grained blocks.In addition, Table 5 differentiates o:?tween the number of functional

blocks initially considered (obtained from recorded protocols), and thenumber of blocksthat were represented in the final intermediate form; these are given incolumns 1 and 3, respectively. Other features used for assessment are the typeof Intermediate Form (IF) used, details specified in IF (i.e., detail in specifyinglinks between blocks), and verbal evidence for the presence of a global plan(i.e., some explication by the participants of the steps that they would followin elaborating the design). Some of the participants did not use an IF and theywere evaluated based on the rest of the design features. The use of highernumber of components, greater detail in IF, higher number of components inIF, and the presence of a g!obal plan implied higher levels of expertise.However, in assigning an overall rating of observed competence at thefunction level (column 6), we focused on the coverage and detail of the IF.

Table 6 shows the evaluations of each participant at the transitionlevel. The features used were the details expressed in decomposition and thehandling of interactions in this level. In general, the more fine-grained thedecomposition and the better the expression of interactions between modulesexpressed in the form of signals, then the greater the ease with which designshould proceed through the implementation. The better the participantsperformed along these two dimensions the greater was their rated level ofexpertise. Column 4 in the table provides overall ratings.

Table 7 shows the evaluations of each participant at theimplementation level. The features used for assessment are the use of higher-level components, use of custom components, types of interaction checksmade, and the strategy employed in problem resolution. This table does notgive an overall rating, but we determined that the use of high level andcustom components demonstrated greater expertise, as did global versus localinteraction checks. In addition, most participants used patching to correct forinteractions.

Product Characteristics. We also characterized participants in terms ofthe completeness and correctness of their final designs. Table 8 gives aqualitative score for the completeness of each participant's finalimplementation (i.e., what proportion of the specified functionality did thefinal design cover) and a score for correctness (i.e., the proportion of specifiedsubfunctionalities correctly implemented in the final solution). Table 78 alsoclassifies participants on a four-point scale, on the basis of the combinedcompleteness and correctness ratings. Level 1 scores reflected minimalcompleteness and correctness in final implementation. Level 2 scoresreflected medium completeness and correctness. Level 3 participants scoredhigh on one dimension and medium on the other. One participant wasclassified at Level 4, and scored high on both dimensions of completeness and


correctness. We left participant 7 unclassified, because he took a radicallydifferent approach to the controller design than other participants. P7employed a programmable logic array in solving the problem, and wasstopped early by the experimenter prior to full implementation.

Integrating Process and Product Characteristics. The product rating andthe previously-described process characteristics are generally consistent withone another. An exception to this was P3 who performed well at the functionand transition levels. This exception would have gone unnoticed had weonly looked at product scores. We were concerned that other importantdifferences in design activity would be masked if we focused solely on theproduct measure. We developed a more integrated rating scheme consistingof four levels as follows.

Novices showed little proficiency in complex design. Novicesscored low in completeness and correctness, and had low functionaland transitional scores as well.Average designers scored medium for completeness and correctnessand demonstrated average performance' at the functional andtransitional level, which is indicative of some understanding of theproblem and a strategy-level implementations.Above average designers demonstrated good understanding ofcomplex design in formulating global plans and applying strategies,but did not perform as well as experts in decomposition andhandling of interactions at the transition level. Their averagecompleteness and correctness was above medium.The expert designer had a very good understanding of design. Thisindividual scored high on both dimensions of completeness andcorrectness, and demonstrated mastery at the functional andtransitional levels. The key to the expert's success can to a largeextent be attributed to his ability to handle complexities at thefunction and transition-levels of design.

These categories were exemplified in the descriptions of the designproblem solving behavior of our participants (see Biswas et al., 1994 fordetails), and provide a strong link between process characteristics and thequality of the final product generated by a designer.

Implications for Assessment of ExpertiseThis study made it clear that there was important information to be

gained by looking at how designers went about producing a final design. Forexample, having only looked at products, i.e., the resultant designs, wewould not have known that participants 9 and 10 used a primitive form ofcase-based reasoning and did not use functional decomposition. Likewise, P3'sperformance was due to difficulties at the implementation level, while her

21


performances at the function and transition levels were quite good. Wewould also not have known that P4 attempted to transfer his expertise inanalog circuit design to the digital case. Furthermore, P11's behaviorsuggested the importance of early error checking and evaluation at thefunctional level in obtaining a good design. The information gained from theprocess level descriptions provides a basis for comparisons with otherdomains in which expertise has been described. In addition, a comparison ofthe processes used by the most expert participant in our group with theprocesses of the less expert participants provides a basis for instructionalinterventions.

Design Tool for Complex Circuit DesignAssuming that one is interested in the richer characterization that

results from considering both process and product descriptions in digitalcircuit design, it is important to make the collection of process data moretractable than through the analysis of think-aloud verbal protocols. Towardsthis end, we built a prototype design tool to assist participants in function andintermediate-level design. The design tool has the following components:

problem-description browsing toolhigh-level formalization and simulation toolcircuit-diagram drawing tool, andcircuit-simulator tool

Preliminary testing conducted on participants as well as the analysis ofthe complex problem described above made it clear that to perform moredetailed experiments (cf. James, 1995) would require that the system providethe capability to generate a number of intermediate forms (IFs) and the abilityto simulate these IFs to check the behavior of the designed system. However,the termination of the project did not allow us to pursue development of thedesign assistant tool any further. Future work in this area should be directedtoward the building of function-level design tools and the capture of designactivity traces at this level of design.

Conclusions

The diagnosis of expertise in a complex design task first requires asufficient understanding of sources of expertise-related differences. Thestudies conducted under the auspices of this project suggest some preliminarycharacterizations. In addition, the work demonstrates the feasibility as well ascurrent limitations of using computerized automated diagnosis systems.

The study of problem solving behavior for the simpler, combinationalcircuit provided a characterization of expert-novice differences.


Intermediate and novice problem solvers exhibited a large amountof "step-by-step" local planning. These often occurred at transitionspoints from one algorithmic component to the next. This patternseemed to indicate that the intermediate and novice problemsolvers were somewhat mechanically implementing an algorithmicsolution without reflecting on the consequences and results theywere obtaining as they solved the problem. In contrast, the expertsappeared to be proceeding with fewer difficulties and to have a inmind a plan for what they would do next. Although only one expertverbalized a global plan,the other expert showed little of theindecision shown by the intermediates when they completed acomponent.To facilitate improved problem solving by intermediates, a secondstudy required theth to generate and verbalize a global plan prior tobeginning solution. The forced generation of a global plan did notimpact any indices of problem solving. However, expertness of theplan generated was predictive of overall problem solving strategy.The primary differentiation was that more expert planners includedBoolean expressions in their solutions more frequently than lessexpert planners.

We did not obtain the magnitude of differentiation among experts andintermediates that we had expected. We attribute this to two related factors.Combinational circuit design is an extremely routine task. Consequently, theexperts were not in the practice of designing these "by hand" but relied onautomated tools to generate these circuits. Second, errors made early in thesolution process made it quite difficult for all participants to discern therelevant pat erns. Had the experts worked on problems that were closer totheir daily job activities, we suspect differences between them andintermediates would have been greatly magnified.

A second goal of our work with the combinational circuit design taskwas to imp 'ement an automated system for administering and assessingstudents. A major issue was whether computer traces of the design processwould be sufficient to distinguish among levels of expertise in design. Thecomputer traces generated from the Design Assistant tool containedinformation sufficient to reliably discern designers' overall and intermediatesteps strategies. These were the two descriptors of problem solvingperformance that were significantly related to expertness of the generatedplan. We also realized that the design tool was not collecting sufficient designactivity information to determine a number of other important elements ofthe design solution process. Several changes that would ameliorate thisproblem could be rather easily implemented in a "next" version of the tool.

The more complex sequential circuit design task proved to be quitesuccessful in differentiating expert, intermediate, and novice behavior. Thiswas achieved by analysis of the high level strategies the participants employed


at the function, transition, and implementation levels of design. A number ofinteresting points will help characterize assessment of expert-novicedifferences in the future.

Experts tend to focus more on function-level design indecomposing a complex system into sub functionalities, anddefining the interactions between the different functional blocks.This guides design at the transition and implementation levels.Intermediates and novices, in particular, do not seem to put in asmuch effort at the function level, which results in their beingoverwhelmed by the complexity of the problem at the transitionand implementation levels. The end result is erroneous and/orsuboptimal design solutions.Expert-novice assessment is best conducted by studying process dataas opposed to product data (i.e., just the quality of a participant'sfinal solutions). Furthermore, analysis of process data can beeffectively conducted by working at strategy-level operations (i.e.,planning behavior) as opposed to more detailed unit-process levelbehavior.It is hard to characterize expertise: expert-level behavior manifestsitself in multiple ways through the use of multiple combinations ofbasic strategies. The one common thread we observed incharacterizing experts was their systematic attempts and success attackling the harder problems early in the design process, making theerror handling and optimization tasks at the implementation levela much easier task.

Comparing the framework generated and the results produced fromthe two sets of studies, we can claim that we were able to define a sufficientlycomplete framework for characterizing both process and product aspects ofdigital circuit design. Our comparative studies with the computer-basedassessment tool for combinational circuits indicated a number of areas wheremore detailed information was required in the traces to ensure that reliableconclusions could be made about design activity. For the more complexproblems, the design of computer-based tools becomes much more difficult,because of the multiple representations that designers employ in functional-and intermediate-level designs. Furthermore, it is still an open question as towhether the strategic behavior of designers can be derived from computertraces of their design activity.


References

Brown, D. C., & Chandrasekaran, B. (1989). Design problem solving: Knowledgestructures and control strategies. San Mateo, CA: Morgan Kaufrnann.

Chi, M. T.. H., Glaser, R., & Farr, M. J. (1988). The nature of expertise (1st ed.).Hillsdale, NJ: Lawrence Erlbaum Associates.

Frederiksen, N., Glaser, R., Lesgold, A., & Shafto, M. G. (Eds.) (1990). Diagnosticmonitoring of skill and knowledge acquisition. Hillsdale, NJ: LawrenceErlbaum Associates.

Gero, J. S. (1990). Design prototypes: A knowledge representation schema fordesign. AI Magazine, 11(4), 26-36.

Gitomer, D. H. (1988). Individual differences in technical troubleshooting.Machine-Mediated Learning, 1, 111-131.

Goel, V., & Pirolli, P. (1989). Motivating the notion of generic design withininformation processing theory: The design problem space. AI Magazine, 10(1),19-36.

Goel, V., & Pirolli, P. (1992). The structure of design problem spaces. CognitiveScience, 16(3), 395-429.

James, C. M., Goldman, S. R., & Vandermolen, H. (1994). Individual differences inplanning-related activities for simple digital circuit design (Tech. Rep. No. 94-1). Nashville, TN: Vanderbilt University.

Pirolli, P. (1992). Knowledge and processes in design. (ONR Final Report).Arlington, VA: Cognitive and Neural Sciences Division.

Maher, M. L. (1988). HI-RISE: An expert system for preliminary structural design.In M. Rychener (Ed.), Expert systems for engineering design. San Diego, CA:Academic Press.

McDermott, J. (1982). R1: A rule-based configurer of computer systems. ArtificialIntelligence, 19(1), 39-88.

Mitchell, W., & Stiny, G. (1978). The palladian grammar. Environment andPlanning B(5), 5-18.

Mostow, J., Barley, M., & Weinrich, J. (1989). Automatic reuse of design plans.Artificial Intelligence in Engineering, 4(4), 181-196.

Simon, H. A. (1969). The sciences of the artificial. Cambridge, MA: MIT Press.Sriram, D. (1986). Knowledge-based approaches for structural design.

Unpublished doctoral dissertation, Carmigie-Mellon University, Pittsburgh,PA.

Tong, C., & Sriram, D. (1992). Introduction. In C. Tong & D. Sriram (Eds.), Artificialintelligence in engineering design (Vol. 1, pp. 1-53). San Diego, CA: AcademicPress.

Ullman, D. G., Dietterich, T. G., & Stauffer, L. A. (1988). A model of the mechanicaldesign process based on empirical data: A summary. Proc. Intl. Conf onApplications of AI in Engineering, pp. 193-216. San Diego, CA: AcademicPress.

Vandermolen, H., James, C. M., Goldman, S. R., Biswas, G., & Bhuva, B. (1992).


Assessing expertise in simple digital circuits. Proceedings of 4th Midwest Aland Cognitive Science Society Conference, pp. 47-51. Starved Rock State Park,IL.

Yoshikawa, H. (1981). General design theory and a CAD system. In T. Sata & E. A.Warman (Eds.), Man-machine communication in CAD/CAM,pp. 35-53. Proc.of the IFIP Working Group 5.2-5.3. Tokyo.

Zhao, F., & Maher, M. L. (1988). Using analogical reasoning to design buildings.Engineering with Computers, 4, 107-119.


Table 1: Combinational circuit design problems

Problem A, The Subtractor ProblemYou are to design a 1-bit full subtractor. Your circuit will accept one bit

of each operand and an input borrow bit and produce a result bit and anoutput borrow.

In solving this problem keep in mind that we are not interested in howmuch time it will take you. We are interested in seeing a good design and thesteps leading you there. Assume you will have to make a product out of thisdesign. The circuit will have to be implemented in a CMOS chip. Your criteriafor "good design" should be (in order of importance):

1. Minimum cost, that is, minimum number of gates used. Keep inmind that some CMOS gates are more complicated than others. Thegoal is thus actually to minimize the area the chip's layout.

2. Testability

Problem B, The job Skill Problem

A certain placement service evaluates people based on their possession(or lack) of four basic skills: typing, basic computer use, friendliness, and godphone voice. Each applicant is evaluated for Type A and Type B jobs. Beingsuitable for one or both types of jobs results in being added to their files withthe types of jobs the applicant is suitable for. If an applicant is suitable forneither job they are rejected. Type A job skills include applicants who: cannottype, but can use a computer, and have a good phone voice; are friendly, havea good phone voice, and can type; are friendly, can use a computer, but can'ttype; can't use a computer, don't have a good phone voice, aren't friendly, andmay or may not be able to type.Type B job skills include applicants who: can't type, and don't have a goodphone voice, but can use a computer; can type, have a poor phone voice, andare friendly; are friendly, but can't type, and don't have a good phone voice;can't use a computer, aren't friendly, but do have a good phone voice.Ns a designer, create a logic circuits that outputs A and B from four inputs w,x, y, and z.

In solving this problem keep in mind that we are not interested in howmuch time it will take you. We are interested in seeing a good design and thesteps leading you there. Assume you will have to make a product out of thisdesig.i. The circuit will have to be implemented in a CMOS chip. Your criteriafor "good design" should be (in order of importance):

1. Minimum cost, that is, minimum number of gates used. Keep inmind that some CMOS gates are more complicated than others. Thegoal is thus actually to minimize the area of the chip's layout.

2. Testability

2


Table 2: Example global plan

Let me demonstrate for you what I mean by an overall plan for the task ofpreparing a shopping trip.

"Well, let's see. I have to buy food for next week. First I'll have tofigure out how many days next week I'll be eating dinner homeand how many times I'll be packing a lunch for work. I don 'twant to get too much food. I should check the refrigerator andthe cupboards to make sure that I don't buy something that Ialready have. Is there a special occasion coming up next weekwhich would require me to make something specific? If so, then Ineed to get the ingredients for it. I need to think about costsavings. Do I have any relevant coupons? Which store should Igo to? If I'm not in a hurry I can go to Mega Market. If I'm in ahurry it is too far away. Kroger's is closer but more expensive."

So by an overall plan, I mean laying out things that need to be thought about,how to accomplish them and contingencies that need to be considered.


Table 3: Overall solution strategies based on video trace

Overall Strategy Problem A Problem B

Textbook 12 (60%) 1.1 (55%)

Textbook without Boolean 6 (30%) 6 (30%)

Textbook without Grouping 2 (10%) 2 (10%)

Textbook without K-map 0 1 (5%)

29


Table 4: Problem description for complex circuit design study

A small company selling networking supplies is experiencing problemswith one of their IC suppliers and decides to design their own networkcontroller. The type of network the company sells is a proprietary token -5ingarchitecture. The token ring controller will be manufactured as a CMOS chipby an external firm, using 2 micron technology. The network normallyconnects 5 to 100 computers, with each computer containing one networkcontroller. You are asked to design the sender part of ,:he controller. The finaldesign should be a gate level design specification. You are free to use higherlevel modules like shift registers, just specify the implementation of one bitof the register at the gate level.The computers are connected in a ring topology. Each computer has a unique7-bit address. A frame of data is of variable length and consists of thefollowing elements:

1. an 8-bit token, indicating the head of a frame (1000 0000)2. a 1-bit busy signal, indicating whether frame is full or empty (1

full, 0 - empty)3. a 7-bit address identifying the intended receiver4. a 1-bit signal acknowledging reception by the receiver (1 accepter,

0 other)5. a 7-bit address identifying the sender6. an N-byte data packet

The controller has the following lines

1. (RI) Ring in (input)2. (RO) Ring out (output)3. (DI) Data in (input)

4. (DF) Data finished (input)5. (FD) Fetch data (output)

6. (CL) Clock (input)

Connected to the ring (receiving data)Connected to the ring (sending data)Connected to the system; Input fordata to be sentSystem has no more dataController needs next data bit fromsystemSynchronous signal for all computers

When the controller receives the token, it checks the next bit to see ifthe following frame contains data. If it does, the controller does nothing(receiver part checks to see if data is intended for this system, and initiatesreception). If the frame is empty, the local system is allowed to send data if ithas any available (DF = 0). In that case the next 7 bits are set to the address ofthe intended receiver. The following bit is set to 0 (acknowledge signal), andthe next 7 bits are set to the address of the sender. Finally data transmission


begins. When no more data is available (DF = 1) transmission stops. Whilethe system is sending it keeps watching for the return of the token. When itreceives a busy bit following return of the token (busy bit generated by itself) itHAS TO stop sending and reverse the busy bit to 0. Generation of sender andreceiver addresses all happens externally on the Data In (DI) line. NOTE: Youare responsible for two outputs (Ring Out and Fetch Data) derived from fourinputs (Ring In, Data In, Data Finished, and Clock); see the figure on theprevious page for explanations of these signals.

Additional Constraints:Minimum buffer size per controller: 8 bitsMaximum buffer size per controller: 16 bitsSpeed of operation: 10 Mbps

Please do not worry about the following aspects:generation and synchronization of clock signalspower down situations in any of the computers on the ringconflicts between data and marker bit patternsreception of packetsloss of tokenno acknowledgement from receiverlimited buffer capacity of the net

3 1


Table 5: Evaluation of function levela

Partici-pant

Number ofComponents

IF Number inIF

Detail inIF

GlobalPlan

TotalScore

P1 3 none n/a na/ No 1

P2 3 Rough State Machine 7 Low No 2

P3 6 Flowchart 6 High No 4

P4 5 none n/a n/a No 1

P5 3 Flowchart 3 Low No 2

P6 6 Flowchart 4 Medium No 2

P7 - Data Flow Diagram 6 Medium No 4

P8 6 Informal State Machine 5 Medium Yes 3

P9 5 none n/a n/a No 1

P10 7 none n/a n/a Yes 1

P11 7 State Machine 6 High Yes 4

aRatings were assigned according to the following criteria: (1) No IF; 2 = lowor medium coverage; low or medium detail; 3 = medium coverage; mediumdetail; and 4 = high coverage; high detail.

32


Table 6: Evaluation of transition levela

Parti-cipant

Core Component Detail inDecomposition

Use of Signals TotalScore

P1 Switch High Low 2

P2 Shift Register Low Low 1

P3 Abstract Controller High Medium 3

P4 Shift Register Medium Medium 2

P5 System State Flop High Medium 3

P6 Shift Register High Low 2

P7 ROM Sequencer -

P8 Shift Register Medium Medium Z.-

P9 Shift Register Medium Low 2

P10 Shift Register Medium High 3

P11 High Level Blocks High High 4

aRatings were assigned in the following manner: 1 = Low decomposition; lowuse of signals; 2 = Medium decomposition; less than medium use of signals; 3= High decomposition and/or high use of signals, but not both; and 4 = Highdecomposition and high use of signals.

33


Table 7: Implementational level

Parti-cipant

Strategy High-LevelComponents

.Custom

ComponentsInteraction

Checks

...,Problem

Resolution

P1 Iterative Addition Yes No Local, SomeGlobal

AbandonedFlawed & CorrectParts

P2 Iterative Addition &Refinement

Yes Yes Local (verylittle)

Immediate Patch

P3 n/a Yes Yes none n/a


Yes No Local Immediate Patch


Yes No Global Immediate Patch

P6 Iterative Addition Yes No Local Immediate Patch

P7 Complete BlockLayout

Yes No none n/a

P8 Global Layout;Iterative Refinement

Yes Yes Local w/ a FinalGlobal

Deferment

P9 Iterative Refinement& Addition

Yes Yes Local Immediate Patch

P10 Iterative Refinement& Addition

Yes Yes Local Immcdiate Patch

P11

-

Iterative Refinementon Signal/Blocks

Yes Yes At Block Level Immediate Patch(One Deferment)

34


Table 8: Overall evaluation

Participant Completeness FlawDetection

OverallRating

P1 Low Low 1

P2 Low Low 1

P3 Low none 1

P4 Medium Medium 2

P5 Medium Medium 2

P6 High Medium 3

P7 Medium n/a

P8 Medium High 3

P9 High Medium 3

P10 High Medium 3

P11 High High 4


reads problem1 4states plan

2 4defines inputs/outputs

3

4truth table

Aiiroi.......

k-map Result k-map B-out5 lir lir 11

10 groups B-out6 4, 4 12

rejects as useless Boolean B-out7 ir 13

implements Result

9Boolean Result

implements B-out

16,18,20

connects15

simu ates17,19

verifies sirn. output21

evaluate - accept

14

Figure 1: Example of serial flow diagram


reads problem1 4,

states plan2 lir

defines inputs/outputs3 lir

truth table4

5

k-map for Result

7Boolean eqn. for Result 4 group B-out

110. k-map for B-out4 6

implement Result10

Boolean e n. for B-out9

11). implement b-out11

simulate4 12

verify sim. outputlir 13

evaluate accept

Figure 2: Example of parallel flow diagram

.1,11,-

(Receive anddecode token

(Yes

Send addressof receiver

cSet next bit(ack) to 0

Do nothing(receive part)


(Reset \Busy bit

Figure 3: Flow-chart form: Token ring controller

iDone

t

Dis

trib

utio

n L

ist

Dis

trib

utio

n L

ist

Dr.

Dav

id L

. Ald

erto

n, C

ode

131

Edw

ard

Atk

ins

Dr.

Joh

n B

lack

CD

R R

ober

t Car

ter

Nav

y Pe

rson

nel R

&D

Cen

ter

1370

5 L

akew

ood

Ct.

Tea

cher

s C

olle

ge, B

ox 8

Nav

al M

edic

al R

esea

rch

San

Die

go, C

A 9

2152

-680

0R

ockv

ille,

MD

208

50C

olum

bia

Uni

vers

ityan

d D

evel

opm

ent C

omm

and

525

Wes

t 120

th S

tree

tN

atio

nal N

aval

Med

ical

Ctr

Dr.

Ter

ry A

llard

Dr.

Pat

rici

a B

agge

ttN

ew Y

ork,

NY

100

27B

ldg.

1-T

-11

Cod

e 34

2CS

Scho

ol o

f E

duca

tion

Bet

hesd

a, M

D 2

0889

-560

6O

ffic

e of

Nav

al R

esea

rch

610

E. U

nive

rsity

, Rrn

130

20D

r. D

ebor

ah A

. Boe

hm-D

avis

800

N. Q

uinc

y St

.U

nive

rsity

of

Mic

higa

nD

epar

tmen

t of

Psyc

holo

gyD

r. E

duar

do C

asca

llar

Arl

ingt

on, V

A 2

2217

-566

0A

nn A

rbor

, MI

4810

9-12

59G

eorg

e M

ason

Uni

vers

ityE

duca

tiona

l Tes

ting

Serv

ice

4400

Uni

vers

ity D

rive

Ros

edal

e R

oad

Dr.

Joh

n R

. And

erso

nD

r. W

illia

m M

. Bar

tFa

irfa

x, V

A 2

2030

-444

4Pr

ince

ton,

NJ

0854

1D

epar

tmen

t of

Psyc

holo

gyU

nive

rsity

of

Min

neso

taC

arne

gie-

Mel

lon

Uni

vers

ityD

ept.

of E

duc.

Psy

chol

ogy

Dr.

Ken

neth

R. B

off

Dr.

Dav

ida

Cha

rney

Sche

n le

y Pa

rk33

0 B

urto

n H

all

AL

/CFH

Eng

lish

Dep

artm

ent

Pitts

burg

h, P

A 1

5213

178

Pills

bury

Dr.

, S.E

.W

righ

t-Pa

tteis

on A

FBPe

nn S

tate

Uni

vers

ityM

inne

apol

is, M

N 5

5455

OH

454

33-6

573

Uni

vers

ity P

ark,

PA

168

02D

r. N

ancy

S. A

nder

son

Dep

artm

ent o

f Ps

ycho

logy

Leo

Bel

trac

chi

Dr.

J. C

. Bou

drea

uxD

r. M

iche

lene

Chi

Uni

vers

ity o

f M

aryl

and

Uni

ted

Stat

es N

ucle

arM

anuf

actu

ring

Eng

inee

ring

Lab

Lea

rnin

g R

& D

Cen

ter

Col

lege

Par

k, M

D 2

0742

Reg

ulat

ory

Com

mis

sion

Nat

iona

l Ins

titut

e of

Uni

vers

ity o

f Pi

ttsbu

rgh

Was

hing

ton

DC

205

55St

anda

rds

and

Tec

hnol

ogy

3939

O'H

ara

Stre

etD

r. T

hom

as H

. And

erso

nG

aith

ersb

urg,

MD

208

99Pi

ttsbu

rgh,

PA

152

60C

ente

r fo

r th

e St

udy

of R

eadi

ngD

r. W

illia

m 0

. Ber

ry17

4 C

hild

ren'

s R

esea

rch

Cen

ter

Dir

ecto

r of

Lif

e an

dD

r. R

icha

rd L

. Bra

nch

Dr.

Sus

an C

hipm

an51

Ger

ty D

rive

Env

iron

men

tal S

cien

ces

HQ

, USM

EPC

OM

/ME

PCT

Cog

nitiv

e Sc

ienc

e Pr

ogra

mC

ham

paig

n, I

L 6

1820

AFO

SR/N

L, N

I, B

ldg.

410

2500

Gre

en B

ay R

oad

Off

ice

of N

aval

Res

earc

hB

ollin

g A

FB, D

C 2

0332

-644

8N

orth

Chi

cago

, IL

600

6480

0 N

. Qui

ncy

St.,

Cod

e 34

2CS

Dr.

Ste

phen

J. A

ndri

ole,

Cha

irm

anA

rlin

gton

, VA

222

17-5

660

Col

lege

of

Info

rmat

ion

Stud

ies

Dr.

Tho

mas

G. B

ever

Dr.

Rob

ert B

reau

xD

rexe

l Uni

vers

ityD

epar

tmen

t of

Psyc

holo

gyC

ode

252

Dr.

Ray

mon

d E

. Chr

ista

lPh

ilade

lphi

a, P

A 1

9104

Uni

vers

ity o

f R

oche

ster

Nav

al T

rain

ing

Syst

ems

Cen

ter

UE

S L

AM

P Sc

ienc

e A

dvis

orR

iver

Sta

tion

Orl

ando

, FL

328

26-3

224

AL

/HR

MIL

Prof

. Joh

n A

nnet

tR

oche

ster

, NY

146

27B

rook

s A

FB, T

X 7

8235

Uni

vers

ity o

f W

arw

ick

Dr.

Ann

Bro

wn

Dep

artm

ent o

f Ps

ycho

logy

Dr.

Men

ucha

Bir

enba

umG

radu

ate

Scho

ol o

f E

duca

tion

Dr.

Deb

orah

Cla

man

Cov

entr

y C

V4

Edu

catio

nal T

estin

g Se

rvic

eU

nive

rsity

of

Cal

ifor

nia

Nat

iona

l Ins

titut

e fo

r A

ging

EN

GL

AN

DPr

ince

ton,

NJ

0854

1E

MST

-453

3 T

olm

an H

all

Bld

g. 3

1, R

oom

5C

-35

Ber

kele

y, C

A 9

4720

9000

Roc

;ill

e Pi

keD

r. G

rego

ry A

nrig

Dr.

Wer

ner

P. B

irke

Bet

hesd

a, iv

1D 2

0892

Edu

catio

nal T

estin

g Se

rvic

ePe

rson

alst

amm

amt d

er B

unde

sweh

rD

r. P

at C

arpe

nter

Prin

ceto

n, N

J 08

541

Kol

ner

Stra

sse

262

Car

negi

e-M

ello

n U

nive

rsity

Dr.

Will

iam

J. C

lanc

eyD

-500

0 K

oeln

90

Dep

artm

ent o

f Ps

ycho

logy

Inst

itute

for

Res

earc

h on

Lea

rnin

gT

echn

ical

Dir

ecto

r, A

R1

FED

ER

AL

RE

PUB

LIC

OF

GE

RM

AN

YPi

ttsbu

rgh,

PA

152

1325

50 H

anov

er S

tree

t50

01 E

isen

how

er A

venu

ePa

lo A

lto, C

A 9

4304

Ale

xand

ria,

VA

223

33

3 3

BE

ST C

OPY

AV

AIL

AB

LE

4 0

Dis

trib

utio

n L

ist

Dis

trib

utio

n L

ist

Chi

ef o

f N

aval

Edu

catio

n an

dD

r. A

lber

t T. C

orbe

ttD

efen

se T

echn

ical

Inf

orm

atio

n C

ente

rD

r. M

arth

a E

vens

Tra

inin

g (N

-5)

Dep

artm

ent o

f Ps

ycho

logy

DT

1C/D

DA

-2D

ept.

of C

ompu

ter

Scie

nce

NA

S Pe

nsac

ola,

FL

325

08C

arne

gie-

Mel

lon

Uni

vers

ityC

amer

on S

tatio

n, B

ldg

5Il

linoi

s In

stitu

te o

f T

echn

olog

yPi

ttsbu

rgh,

PA

152

13A

lexa

ndri

a, V

A 2

2314

10 W

est 3

Ist S

tree

tD

r. P

aul C

obb

(4 /C

opie

s)C

hica

go, I

L 6

0616

104C

May

born

Bui

ldin

gD

r. M

icha

el C

owen

Box

330

Pea

body

Col

lege

Cod

e 14

2M

r. D

avid

DuB

ois

Dr.

Lor

rain

e D

. Eyd

eV

ande

rbilt

Uni

vers

ityN

avy

Pers

onne

l R&

D C

ente

rPe

rson

nel D

ecis

ions

Res

earc

hU

S O

ffic

e of

Per

sonn

el M

anag

emen

tN

ashv

ille,

TN

372

03Sa

n D

iego

, CA

921

52-6

800

Inst

itute

sO

ffic

e of

Per

sonn

el R

esea

rch

and

43 M

ain

Stre

et, S

ED

evel

opm

ent

Dr.

Rod

ney

Coc

king

Dr.

Ken

neth

B. C

ross

Riv

erpl

ace,

Sui

te 4

0519

00 E

St.,

NW

NIM

H, B

asic

Beh

avio

r an

dA

naca

pa S

cien

ces,

Inc

.M

inne

apol

is, M

N 5

5414

Was

hing

ton,

DC

204

15C

ogni

tive

Scie

nce

Res

earc

hP.

O. B

ox 5

1956

00 F

ishe

rs L

ane,

Rrn

11C

-10

Sant

a B

arba

ra, C

A 9

3102

Dr.

Ric

hard

Dur

anD

r. F

ranc

o Fa

ina

Park

law

n B

uild

ing

Gra

duat

e Sc

hool

of

Edu

catio

nD

iret

tore

Gen

eral

e L

EV

AD

IFE

Roc

kvill

e, M

D 2

3857

CT

B/M

cGra

w-H

ill L

ibra

ryU

nive

rsity

of

Cal

ifor

nia

Piaz

zale

K. A

dena

uer,

325

00 G

arde

n R

oad

. San

ta B

arba

ra, C

A 9

3106

0014

4 R

OM

A E

UR

Dir

ecto

r, L

ife

Scie

nces

Mon

tere

y, C

A 9

3940

-538

0IT

AL

YO

ffic

e of

Nav

al R

esea

rch

Dr.

Nan

cy E

ldre

dge

Cod

e 11

4D

r. C

harl

es E

. Dav

isC

olle

ge o

f E

duca

tion

Dr.

Jea

n-C

laud

e Fa

lmag

neA

rlin

gton

, VA

222

17-5

000

Edu

catio

nal T

estin

g Se

rvic

eD

ivis

ion

of S

peci

al E

duca

tion

Irvi

ne R

esea

rch

Uni

t in

Mai

l Sto

p 22

-TT

he U

nive

rsity

of

Ari

zona

Mat

hem

atic

al d

r B

ehav

iora

l Sci

ence

sD

irec

tor,

Cog

nitiv

e an

dPr

ince

ton,

NJ

0854

1T

ucso

n, A

Z 8

5721

Uni

vers

ity o

f C

alif

orni

aN

eura

l Sci

ence

s, C

ode

1142

Irvi

ne, C

A 9

2717

Off

ice

of N

aval

Res

earc

hD

r. G

eory

Del

acot

eD

r. J

ohn

Elli

sA

rlin

gton

, VA

222

17-5

000

Exp

lora

tori

umN

avy

Pers

onne

l R&

D C

ente

rD

r. B

eatr

ice

J. F

arr

3601

Lyo

n St

reet

Cod

e 15

Arm

y R

esea

rch

Inst

itute

Dir

ecto

rSa

n Fr

anci

sco,

CA

941

23Sa

n D

iego

, CA

921

52-6

800

PER

I-IC

Tra

inin

g Sy

stem

s D

epar

tmen

t50

01 E

isen

how

er A

venu

eC

ode

15A

Dr.

Sha

ron

Der

ryD

r. S

usan

Em

bret

son

Ale

xand

ria,

VA

223

33N

avy

Pers

onne

l R&

D C

ente

rFl

orid

a St

ate

Uni

vers

ityU

nive

rsity

of

Kan

sas

San

Die

go, C

A 9

2152

-680

0D

epar

tmen

t of

Psyc

holo

gyPs

ycho

logy

Dep

artm

ent

Dr.

Mar

shal

l J. F

arr

Tal

laha

ssee

, FL

323

0642

6 Fr

aser

Farr

-Sig

ht C

o.L

ibra

ry, C

ode

231

Law

renc

e, K

S 66

045

2520

Nor

th V

erno

n St

reet

Nav

y Pe

rson

nel R

&D

Cen

ter

Dr.

Ste

phan

ie D

oane

Arl

ingt

on, V

A 2

2207

San

Die

go, C

A 9

2152

-580

0U

nive

rsity

of

Illin

ois

Dr.

Sus

an E

pste

inD

epar

tmen

t of

Psyc

holo

gy14

4 S.

Mou

ntai

n A

venu

eD

r. P

at-A

ntho

ny F

rede

rico

Dr.

Mag

da C

olbe

rg60

3 E

ast D

anie

l Str

eet

Mon

tcla

ir, N

J 07

042

Cod

e 13

, NPR

DC

Off

ice

of P

erso

nnel

Man

agem

ent

Cha

mpa

ign,

IL

618

2053

335

Bry

ne R

oad

1900

E S

tree

t, N

.W.

ER

IC F

acili

ty-A

cqui

sitio

nsSa

n D

iego

, CA

921

52-7

250

Was

hing

ton,

DC

204

15-0

001

Dr.

Mic

hael

Dri

lling

s13

01 P

icca

rd D

rive

, Sui

te 3

00B

asic

Res

earc

h O

ffic

eR

ockv

ille,

MD

208

50-4

305

Dr.

J. D

. Fle

tche

rC

omm

andi

ng O

ffic

erA

rmy

Res

earc

h In

stitu

teIn

stitu

te f

or D

efen

se A

naly

ses

Nav

al R

esea

rch

Lab

orat

ory

5001

Eis

enho

wer

Ave

nue

1801

N. B

eaur

egar

d St

.C

ode

4827

Ale

xand

ra, V

A 2

2333

-560

0A

lexa

ndri

a, V

A 2

2311

Was

hing

ton,

DC

203

75-5

000 41

42

Dis

trib

utio

n L

ist

Dr.

Law

renc

e T

. Fra

seE

xecu

tive

Dir

ecto

rD

ivis

ion

of C

ogni

tive

and

Inst

ruct

iona

l Sci

ence

Edu

catio

nal T

estin

g Se

rvic

ePr

ince

ton,

NJ

0854

1

Dr.

Car

l H. F

rede

riks

enD

ept.

of E

duca

tiona

l Psy

chol

ogy

McG

ill U

nive

rsity

3700

McT

avis

h St

reet

Mon

trea

l, Q

uebe

cC

AN

AD

A H

3A 1

Y2

Dr.

Nor

man

Fre

deri

ksen

Edu

catio

nal T

estin

g Se

rvic

e(0

5-R

)Pr

ince

ton,

NJ

0854

1

Dr.

Joh

n D

. E. G

abri

eli

Dep

artm

ent o

f Ps

ycho

logy

Jord

an H

all

Stan

ford

Uni

vers

itySt

anfo

rd, C

A 9

4305

Dr.

Jac

k J.

Gel

fand

Dep

artm

ent o

f Ps

cyho

logy

Prin

ceto

n U

nive

rsity

Prin

ceto

n, N

J 08

544-

1010

Cha

ir, D

epar

tmen

t of

Com

pute

r Sc

ienc

eG

eorg

e M

ason

Uni

vers

ityFa

irfa

x, V

A 2

2030

Dr.

Ala

n S.

Gev

ins

EE

G S

yste

ms

Lab

orat

ory

51 F

eder

al S

tree

t, Su

ite 4

01Sa

n Fr

anci

sco,

CA

941

07

Dr.

Hel

en G

igle

yN

aval

Res

earc

h L

ab.,

Cod

e 55

3045

55 O

verl

ook

Ave

nue,

S. W

.W

ashi

ngto

n, D

C 2

0375

-500

0

43

Dr.

Her

bert

Gin

sbur

gB

ox 1

84T

each

ers

Col

lege

Col

umbi

a U

nive

rsity

525

Wes

t 121

st S

tree

tN

ew Y

ork,

NY

100

27

Dr.

Dre

w G

itom

erE

duca

tiona

l Tes

ting

Serv

ice

Prin

ceto

n, N

J 08

541

Dr.

Rob

ert G

lase

rL

earn

ing

Res

earc

h&

Dev

elop

men

t Cen

ter

Uni

vers

ity o

f Pi

ttsbu

rgh

3939

O'H

ara

Stre

etPi

ttsbu

rgh,

PA

152

60

Dr.

Sam

Glu

cksb

erg

Dep

artm

ent o

f Ps

ycho

logy

Prin

ceto

n U

nive

rsity

Prin

ceto

n, N

J 08

544-

1010

Prof

. Jos

eph

Gog

uen

PRG

, Uni

v. o

f O

xfor

d11

Keb

le R

oad

Oxf

ord

OX

13Q

DU

NIT

ED

KIN

GD

OM

Dr.

Pau

l E. G

old

Uni

vers

ity o

f V

irgi

nia

Dep

artm

ent o

f Ps

ycho

logy

Cha

rlot

tesv

ille,

VA

229

03

Dr.

Sus

an R

. Gol

dman

Peab

ody

Col

lege

, Box

45

Van

derb

ilt U

nive

rsity

Nas

hvill

e, T

N 3

7203

Dr.

Tim

othy

Gol

dsm

ithD

epar

tmen

t of

Psyc

holo

gyU

nive

rsity

of

New

Mex

ico

Alb

uque

rque

, NM

871

31

Dis

trib

utio

n L

ist

Dr.

She

rrie

Got

tA

FHR

L/M

OM

JB

rook

s A

FB, T

X 7

8235

-560

1

Dr.

T. G

ovin

dara

jG

eorg

ia I

nstit

ute

of T

echn

olog

ySc

hool

of

Indu

stri

alan

d Sy

stem

s E

ngin

eeri

ngA

tlant

a, G

A 3

0332

-020

5

Dr.

Mar

ilyn

K. G

owin

gO

ffic

e of

Per

sonn

el R

&D

1900

E S

t., N

W, R

oom

646

2O

ffic

e of

Per

sonn

el M

anag

emen

tW

ashi

ngto

n, D

C 2

0415

Jord

an G

rafm

an, P

h.D

.C

hief

, Cog

nitiv

e N

euro

scie

nce

Sect

ion,

Med

ical

Neu

rolo

gyB

ranc

h-N

IND

S, B

ldg

10, R

m. 5

C42

2B

ethe

sda,

MD

208

92

Dr.

Way

ne G

ray

Gra

duat

e Sc

hool

of

Edu

cat,o

nFo

rdha

m U

nive

rsity

113

Wes

t 60t

h St

reet

New

Yor

k, N

Y 1

0023

Dr.

Ber

t Gre

enJo

hns

Hop

kins

Uni

vers

ityD

epar

tmen

t of

Psyc

holo

gyC

harl

es &

34t

h St

reet

Bal

timor

e, M

D 2

1218

Dr.

Jam

es G

. Gre

eno

Scho

ol o

f E

duca

tion

Stan

ford

Uni

vers

ity, R

oom

311

Stan

ford

, CA

943

05

Dr.

Ste

phen

Gro

ssbe

rgC

ente

r fo

r A

dapt

ive

Syst

ems

Roo

m 2

4411

1 C

umm

ingt

on S

tree

tB

osto

n U

nive

rsity

Bos

ton,

MA

022

15

Prof

. Edw

ard

Hae

rtel

Scho

ol o

f E

duca

tion

Stan

ford

Uni

vers

itySt

anfo

rd, C

A 9

4305

-309

6

Mar

ilyn

Hal

pern

, Lib

rari

anB

righ

am L

ibra

ryE

duca

tiona

l Tes

ting

Serv

ice

Car

ter

and

Ros

edal

e R

oads

Prin

ceto

n, N

J 08

541

Dr.

Ste

phen

J. H

anso

nL

earn

ing

& K

now

ledg

eA

cqui

sitio

n R

esea

rch

Siem

ens

Res

earc

h C

ente

r75

5 C

olle

ge R

oad

Eas

tPr

ince

ton,

NJ

0854

0

Dr.

Del

wyn

Ham

isch

Uni

vers

ity o

f Il

linoi

s51

Ger

ty D

rive

Cha

mpa

ign,

IL

618

20

Dr.

Pat

rick

R. H

arri

son

Com

pute

r Sc

ienc

e D

epar

tmen

tU

.S. N

aval

Aca

dem

yA

nnap

olis

, MD

214

02-5

002

Dr.

Way

ne H

arve

yE

duca

tion

Dev

elop

men

t Cen

ter

55 C

hape

l Str

eet

New

ton,

MA

021

60

Dr.

Bar

bara

Hay

es-R

oth

Kno

wle

dge

Syst

ems

Lab

orat

ory

Stan

ford

Uni

vers

ity70

1 W

elch

Roa

d, B

ldg.

CPa

lo A

lto, C

A 9

4304

Dr.

Per

Hel

mer

sen

Uni

vers

ity o

f O

slo

USI

TB

ox 1

059

0316

Osl

o, N

OR

WA

Y

4 4

Dis

trib

utio

n L

ist

Dis

trib

utio

n L

ist

Dr.

N. G

uns

Hoo

fd V

anD

r. M

artin

J. I

ppel

Dr.

Mar

cel J

ust

Dr.

Wal

ter

Kin

tsch

AFD

.SW

OC

ente

r fo

r th

e St

udy

ofC

arne

gie-

Mel

lon

Uni

vers

ityD

epar

tmen

t of

Psyc

holo

gyA

drni

ralit

eit K

r. D

354

Edu

catio

n an

d In

stru

ctio

nD

epar

tmen

t of

Psyc

holo

gyU

nive

rsity

of

Col

orad

oV

an D

er B

urch

laan

31

Lei

den

Uni

vers

itySc

henl

ey P

ark

Bou

lder

, CO

803

09-0

345

Post

Box

207

02.2

500

ES

The

Hag

ueP.

O. B

ox 9

555

Pitts

burg

h, P

A 1

5213

The

NE

TH

ER

LA

ND

S23

00 R

B L

eide

nD

r. S

usan

S. K

irsc

henb

aum

TH

E N

ET

HE

RL

AN

DS

Dr.

Rut

h K

anfe

rC

ode

2212

, Bui

ldin

g 11

71 /

1Pr

of. L

utz

F. H

ornk

eU

nive

rsity

of

Min

neso

taN

aval

Und

erw

ater

Sys

tem

s C

ente

rIn

stitu

t fur

Psy

chol

ogie

Dr.

Rob

ert J

anna

rone

Dep

artm

ent o

f Ps

ycho

logy

New

port

, RI

0284

1R

WT

H A

ache

nE

lec.

and

Com

pute

r E

ng. D

ept.

Elli

ott H

all

Jaeg

erst

rass

e 17

/19

Uni

vers

ity o

f So

uth

Car

olin

a75

E. R

iver

Roa

dD

r. J

anet

L. K

olod

ner

D-5

100

Aac

hen

Col

umbi

a, S

C 2

9208

Min

neap

olis

, MN

554

55G

eorg

ia I

nstit

ute

of T

echn

olog

yW

EST

GE

RM

AN

YC

olle

ge o

f C

ompu

ting

Dr.

Cla

ude

Janv

ier

Dr.

Mic

hael

Kap

lan

Atla

nta,

GA

303

32-0

280

Ms.

Jul

ia S

. Hou

ghC

IRA

DE

, X-7

120

Off

ice

of B

asic

Res

earc

hC

ambr

idge

Uni

vers

ity P

ress

UQ

AM

U.S

. Arm

y R

esea

rch

Inst

itute

Dr.

Syl

van

Kor

nblu

m40

Wes

t 20t

h St

reet

P. 0

. Box

888

8, S

ucc.

A50

01 E

isen

how

er A

venu

eU

nive

rsity

of

Mic

higa

nN

ew Y

ork,

NY

100

11M

ontr

eal,

Que

bec

H3C

3P8

Ale

xand

ria,

VA

223

33-5

600

Men

tal H

ealth

Res

earc

h In

stitu

teC

AN

AD

A20

5 W

asht

enaw

Pla

ceD

r. W

illia

m H

owel

lD

r. W

endy

Kel

logg

Ann

Arb

or, M

I 48

109

Chi

ef S

cien

tist

Dr.

Rob

in J

effr

ies

IBM

T. J

. Wat

son

Res

earc

h C

tr.

AFH

RL

/CA

Hew

lett-

Pack

ard

Lab

orat

orie

s, 1

U-1

7P.

O. B

ox 7

04D

r. S

teph

en K

ossl

ynB

rook

s A

FB, T

X 7

8235

-560

1P.

O. B

ox 1

0490

Yor

ktow

n H

eigh

ts, N

Y 1

0598

Har

vard

Uni

vers

ityPa

lo A

lto, C

A 9

4303

-096

912

36 W

illia

m J

ames

Hal

lD

r. E

va H

udlic

kaD

r. J

.A.S

. Kel

so33

Kir

klan

d St

.B

BN

Lab

orat

orie

sD

r. E

dgar

M. J

ohns

onC

ente

r fo

r C

ompl

ex S

yste

ms

Cam

brid

ge, M

A 0

2138

10 M

oulto

n St

reet

Tec

hnic

al D

irec

tor

Bui

ldin

g M

T 9

Cam

brid

ge, M

A 0

2238

U.S

. Arm

y R

esea

rch

Inst

itute

Flor

ida

Atla

ntic

Uni

vers

ityD

r. K

enne

th K

otov

sky

5001

Eis

enho

wer

Ave

nue

Boc

a R

aton

, FL

334

31D

epar

tmen

t of

Psyc

holo

gyD

r. E

arl H

unt

Ale

xand

ria,

VA

223

33-5

600

Car

negi

e-M

ello

n U

nive

rsity

Dep

t. of

Psy

chol

ogy,

NI-

25C

DR

Rob

ert S

. Ken

nedy

5000

For

bes

Ave

nue

Uni

vers

ity o

f W

ashi

ngto

nD

r. P

eder

Joh

nson

Ess

ex C

orpo

ratio

nPi

ttsbu

rgh,

PA

152

13Se

attle

, WA

981

95D

epar

tmen

t of

Psyc

holo

gy10

40 W

oodc

ock

Roa

dU

nive

rsity

of

New

Mex

ico

Suite

227

Dr.

Ric

hard

J. K

oube

kD

r. J

ack

Hun

ter

Alb

uque

rque

, NM

871

31O

rlan

do, F

L 3

2803

Scho

ol o

f In

dust

rial

Eng

inee

ring

2122

Coo

lidge

Str

eet

Gri

ssom

Hal

lL

ansi

ng, M

I 48

906

Dr.

Dan

iel B

. Jon

esD

r. D

avid

Kie

ras

Purd

ue U

nive

rsity

US

Nuc

lear

Reg

ulat

ory

Com

mis

sion

Tec

hnic

al C

omm

unic

atio

n Pr

ogra

mW

est L

afay

ette

, IN

479

07D

r. G

iorg

io I

ngar

giol

aN

RR

/12

E4

TID

AL

Bld

g., 2

360

Bon

iste

el B

lvd.

Com

pute

r Sc

ienc

e D

epar

tmen

tW

ashi

ngto

n, D

C 2

0555

Uni

vers

ity o

f M

ichi

gan

Dr.

Art

Kra

mer

Tem

ple

Uni

vers

ityA

nn A

rbor

, MI

4810

9-21

08U

niv.

of

Illin

ois

at U

-CPh

ilade

lphi

a, P

A 1

9122

Dr.

Joh

n Jo

nide

sB

eckm

an I

nstit

ute

Dep

artm

ent o

f Ps

ycho

logy

Dr.

Sun

g-H

o K

imA

05 N

. Mat

hew

s A

venu

eU

nive

rsity

of

Mic

higa

nE

duca

tiona

l Tes

ting

Serv

ice

Urb

ana,

IL

618

01A

nn A

rbor

, MI

4810

4Pr

ince

ton,

NJ

0854

1

r JB

ES

T C

OP

Y A

VA

ILA

BLE

46

Dis

trib

utio

n L

ist

Dis

trib

utio

n L

ist

Dr.

Ben

jam

in K

uipe

rsD

r. R

icha

rd L

esh

Dr.

Nad

ine

Mar

tinD

r. A

ndre

w R

. Mol

nar

Uni

vers

ity o

f T

exas

at A

ustin

Edu

catio

nal T

estin

g Se

rvic

eD

epar

tmen

t of

Neu

rolo

gyA

pplic

atio

ns o

f A

dvan

ced

Dep

artm

ent o

f C

ompu

ter

Scie

nces

Prin

ceto

n, N

J 08

541

Cen

ter

for

Cog

nitiv

e N

euro

scie

nce

Tec

hnol

ogie

s, R

m. 6

35A

ustin

, Tex

as 7

8712

Tem

ple

Uni

vers

ity S

choo

l of

Med

icin

eN

atio

nal S

cien

ce F

ound

atio

nD

r. M

icha

el L

evin

e34

01 N

orth

Bro

ad S

tree

tW

ashi

ngto

n, D

C 2

0550

Dr.

Mic

hael

Kup

erst

ein

Edu

catio

nal P

sych

olog

yPh

ilade

lphi

a, P

A 1

9140

Sym

bus

Tec

hnol

ogy

210

Edu

catio

n B

ldg.

Dr.

Ben

B. M

orga

n, J

r.32

5 H

arva

rd S

tree

t13

10 S

outh

Six

th S

tree

tD

r. J

osep

h M

cLac

hlan

Dep

artm

ent o

f Ps

ycho

logy

Suite

211

Uni

vers

ity o

f IL

at U

rban

:1-C

ham

paig

nN

avy

Pers

onne

l Res

earc

hU

nive

rsity

of

Cen

tral

Flo

rida

Bro

oklin

e, M

A 0

2146

Cha

mpa

ign,

IL

618

20-6

990

and

Dev

elop

men

t Cen

ter

Orl

ando

, FL

328

16-0

150

Cod

e 14

Dr.

Pat

rick

Kyl

lone

nL

ogic

on I

nc. (

Attn

: Lib

rary

)Sa

n D

iego

, CA

921

52-6

800

Dr.

Ran

dy M

umaw

AFH

RL

/MO

EL

Tac

tical

and

Tra

inin

g Sy

stem

s D

iv.

Hum

an S

cien

ces

Bro

oks

AFB

, TX

782

35P.

O. B

ox 8

5158

Dr.

Mic

hael

Mc

Nee

seW

estin

ghou

se S

cien

ceSa

n D

iego

, CA

921

38-5

158

DE

T-1

, AL

\ C

FHI

& T

echn

olog

y C

tr.

Dr.

M. D

iane

Lan

gsto

nB

LD

G 2

4813

10 B

eula

h R

oad

ICL

Nor

th A

mer

ica

Prof

. Dav

id F

. Loh

man

Wri

ght-

Patte

rson

AFB

, OH

454

32Pi

ttsbu

rgh,

PA

152

3511

490

Com

mer

ce P

ark

Dri

veC

olle

ge o

f E

duca

tion

Res

ton,

VA

220

91U

nive

rsity

of

Iow

aD

r. A

lan

Mey

row

itzD

r. A

llen

Mun

roIo

wa

City

, IA

522

42N

aval

Res

earc

h L

abor

ator

yB

ehav

iora

l Tec

hnol

ogy

Dr.

Mar

cy L

ansm

anC

ode

5510

Lab

orat

orie

s -

USC

Uni

vers

ity o

f N

orth

Car

olin

aV

ern

M. M

alec

4555

Ove

rloo

k A

ve.,

SW25

0 N

. Har

bor

Dr.

, Sui

te 3

09D

ept.

of C

ompu

ter

Scie

nce

NPR

DC

, Cod

e 14

2W

ashi

ngto

n, D

C 2

0375

-500

0R

edon

do B

each

, CA

902

77C

B #

3175

San

Die

go, C

A 9

2152

-680

0C

hape

l Hill

, NC

275

99D

r. R

ysza

rd S

. Mic

hals

kiA

cade

mic

Pro

gs. &

Res

earc

h B

ranc

hD

r. S

andr

a P.

Mar

shal

lC

ente

r fo

r A

rtif

icia

l Int

ellig

ence

Nav

al T

echn

ical

Tra

inin

g C

omm

and

Ric

hard

Lan

term

anD

ept.

of P

sych

olog

yG

eorg

e M

ason

Uni

vers

ityC

ode

N-6

2C

omm

anda

nt (

G-P

WP)

San

Die

go S

tate

Uni

vers

itySc

ienc

e an

d T

ech

II, R

m.4

11N

AS

Mem

phis

(75

)U

S C

oast

Gua

rdSa

n D

iego

, CA

921

8244

00 U

nive

rsity

Dri

veM

illin

gton

, TN

308

5421

00 S

econ

d St

., SW

Fair

fax,

VA

220

30-4

444

Was

hing

ton,

DC

205

93-0

001

Dr.

Rob

ert W

. Law

ler

Dr.

Cle

ssen

J. M

artin

Hea

d, D

oD C

oord

inat

or,

Rec

rutin

g Pl

ans

and

Prog

ram

sD

r. V

ittor

io M

idor

oC

NR

-Ist

ituto

Tec

nolo

gie

Did

attic

he

Dep

uty

Dir

ecto

r M

anpo

wer

,Pe

rson

nel a

nd T

rain

ing

Div

.N

aval

Sea

Sys

tem

s C

omm

and

Mat

thew

s 11

8B

ranc

h C

ode:

PE

RS

2FF/

234

Via

All'

Ope

ra P

ia 1

1A

TT

N: C

ode

04M

P 51

1Pu

rdue

Uni

vers

ityN

avy

Ann

ex, R

oom

283

2G

EN

OV

A-I

TA

LIA

161

45W

ashi

ngto

n, D

C 2

0362

Wes

t Laf

ayet

te, I

N 4

7907

Was

hing

ton,

DC

203

50D

r. M

ortim

er M

ishk

inM

r. J

. Nel

isse

nD

r. P

aul E

. Leh

ner

Dr.

Eliz

abet

h M

artin

Lab

orat

ory

of N

euro

psyc

holo

gyT

wen

te U

nive

rsity

Dep

artm

ent o

f In

form

atio

nA

L/H

RA

, Sto

p 44

Nat

iona

l Ins

titut

e of

Men

tal H

eatlh

Fac.

Bib

l. T

oege

past

e O

nder

wys

kund

eSy

stem

s &

Eng

inee

ring

Will

iam

s A

FB, A

Z 8

5240

9000

Roc

kvill

e Pi

ke, B

ldg.

9, #

1N10

7P.

O. B

ox 2

17G

eorg

e M

ason

Uni

vers

ityB

ethe

sda,

MD

208

9275

00 A

E E

nsch

ede

4400

Uni

vers

ity D

rive

The

NE

TH

ER

LA

ND

SFa

irfa

x, V

A 2

2030

-444

4D

r. R

ober

t Mis

levy

Edu

catio

nal T

estin

g Se

rvic

ePr

ince

ton,

NJ

0854

1

BE

ST

CO

PY

AV

AIL

AB

LE

Dis

trib

utio

n L

ist

Dis

trib

utio

n L

ist

Dir

ecto

r, I

nsh.

Dev

el. a

ndD

r. J

udith

Rei

tman

Ols

onD

r. R

oy P

eaD

r. P

eter

Pir

olli

Edu

c. P

rog.

Sot

. Dep

t.G

radu

ate

Scho

ol o

f B

usin

ess

Inst

itute

for

the

Lea

rnin

g Sc

ienc

esSc

hool

of

Edu

catio

nN

aval

Edu

c. &

ing.

Pro

g.U

nive

rsity

of

Mic

higa

nN

orth

wes

tern

Uni

vers

ityU

nive

rsity

of

Cal

ifor

nia

Mgt

. Spt

. Act

ivity

(N

ET

PMSA

)A

nn A

rbor

, MI

4810

9-12

3418

90 M

aple

Ave

nue

Ber

kele

y, C

A 9

4720

Pens

acol

a, F

L 3

2509

Eva

nsto

n, I

L 6

0201

Off

ice

of N

aval

Res

earc

hD

r. M

arth

a Po

ison

Dr.

Pau

l Nic

hols

Res

iden

t Rep

rese

ntat

ive

G. P

elsm

aker

sD

epar

tmen

t of

Psyc

holo

gyA

mer

ican

Col

lege

Tes

ting

Geo

rgia

Ins

titut

e of

Tec

hnol

ogy

Rue

Fri

tz T

ouss

aint

47

Uni

vers

ity o

f C

olor

ado

2201

N D

odge

Str

eet

206

O'K

eefe

Bui

ldin

gG

enda

rmer

ie R

SPB

ould

er, C

O 8

0309

-034

4PO

Box

168

Atla

nta,

GA

303

32-0

490

1050

Bru

xelle

sIo

wa

City

, IA

522

43B

EL

GIU

MD

r. P

eter

Poi

son

Dr.

Ray

mon

d S.

Nic

kers

onO

ffic

e of

Nav

al R

esea

rch,

Cod

e 34

2CS

Dr.

Ray

S. P

erez

Uni

vers

ity o

f C

olor

ado

Dep

artm

ent o

f Ps

ycho

logy

5 G

leas

on R

oad

800

N. Q

uinc

y St

reet

AR

I (P

ER

I-II

)B

ould

er, C

O 8

0309

-034

4B

edfo

rd, M

A 0

1730

Arl

ingt

on, V

A 2

2217

-566

050

01 E

isen

how

er A

venu

e(6

Cop

ies)

Ale

xand

ria,

VA

223

33D

r. J

osep

h Ps

otka

Dr.

Don

ald

A. N

orm

anA

TT

N: P

ER

I-IC

Dep

artm

ent o

f C

ogni

tive

Scie

nce

Ass

ista

nt f

or T

rain

ing

Tec

hnol

ogy

C.V

. (M

D)

Dr.

Ant

onio

Per

iA

rmy

Res

earc

h In

stitu

teU

nive

rsity

of

Cal

ifor

nia

and

Hum

an F

acto

rsC

apta

in I

TN

MC

5001

Eis

enho

wer

Ave

.L

a Jo

lla, C

A 9

2093

-051

5O

ffic

e of

the

DC

NO

(MPT

) (0

p-11

E)

Mar

iper

s U

.D.G

. 3 S

ezA

iexa

ndri

a, V

A 2

2333

-560

0D

epar

tmen

t of

the

Nav

yM

INIS

TE

RO

DIF

ESA

- M

AR

INA

Dir

ecto

r, F

leet

Lia

ison

Off

ice

Was

hing

ton,

DC

203

50-2

000

0010

0 R

OM

A -

IT

AL

YPs

yc I

nfo

- C

D a

nd M

NPR

DC

(C

ode

01F)

Am

eric

an P

sych

olog

ical

Ass

oc.

San

Die

go, C

A 9

2152

-680

0D

r. J

udith

Ora

sanu

Dr.

Nan

cy N

. Per

ry12

00 U

hle

Stre

etM

ail S

top

239-

1N

aval

Edu

catio

n an

d T

rain

ing

Arl

ingt

on, V

A 2

2201

Dir

ecto

rN

ASA

Am

es R

esea

rch

Cen

ter

Prog

ram

Sup

port

Act

ivity

Tra

inin

g Sy

stem

s D

epar

tmen

tM

offe

tt Fi

eld,

CA

940

35C

ode-

047,

Bui

ldin

g 24

35D

r. M

ark

D. R

ecka

seN

PRD

C (

Cod

e 14

)Pe

nsac

ola,

FL

325

09-5

000

AC

TSa

n D

iego

, CA

921

52-6

800

Dr.

Eve

rett

Palm

erP.

0. B

ox 1

68M

ail S

top

262-

4C

DR

Fra

nk C

. Pet

hoIo

wa

City

, IA

522

43L

ibra

ry, N

PRD

CN

ASA

-Am

es R

esea

rch

Cen

ter

Nav

al P

ostg

radu

ate

Scho

olC

ode

041

Mof

fett

Fiel

d, C

A 9

4035

Cod

e O

R/P

ED

r. L

ynne

Red

erSa

n D

iego

, CA

921

52-6

800

Mon

tere

y, C

A 9

3943

Dep

artm

ent o

f Ps

ycho

logy

Dr.

Okc

hoon

Par

kC

arne

gie-

Mel

lon

Uni

vers

ityD

r. H

arol

d F.

O'N

eil,

Jr.

Arm

y R

esea

rch

Inst

itute

Dep

t. of

Adm

inis

trat

ive

Scie

nces

Sche

nley

Par

kSc

hool

of

Edu

catio

n W

PH 6

00PE

RI-

2C

ode

54Pi

ttsbu

rgh,

PA

152

13D

epar

tmen

t of

Edu

catio

nal

5001

Eis

enho

wer

Ave

nue

Nav

al P

ostg

radu

ate

Scho

olPs

ycho

logy

& T

echn

olog

yA

lexa

ndri

a, V

A 2

2333

Mon

tere

y, C

A 9

3943

-502

6D

r. J

. Wes

ley

Reg

ian

Uni

vers

ity o

f So

uthe

rn C

alif

orni

aA

rm9t

rone

l.ab

orat

ory

Los

Ang

eles

, CA

900

89-0

031

Way

ne M

. Pat

ienc

eD

r. E

lizab

eth

A. P

help

sA

FHR

L/I

D1

Am

eric

an C

ounc

il on

Edu

catio

nD

epar

tmen

t of

Psyc

holo

gyB

rook

s A

FB, T

X 7

8235

-500

0D

r. S

te I

lan

Ohl

sson

GE

D T

estin

g Se

rvic

e, S

uite

20

New

Sch

ool f

or S

ocia

l Res

earc

hL

earn

ing

R &

D C

ente

rO

ne D

upon

t Cir

cle,

NW

65 F

ifth

Ave

nue

Dr.

Fre

d R

eif

Uni

vers

ity o

f Pi

ttsbu

rgh

Was

hing

ton,

DC

200

36N

ew Y

ork,

NY

100

03C

DE

C, S

mith

Hal

lPi

ttsbu

rgh,

PA

152

60C

arne

gie-

Mel

lon

Uni

vers

ityPi

ttsbu

rg, P

A 1

5213

4950

Dis

trib

utio

n L

ist

Dr.

Cha

rles

M. R

eige

luth

Cha

irm

an, I

nstr

uctio

nal

Syst

ems

Tec

hnol

ogy

Scho

ol o

f E

duca

tion,

Rrn

. 210

Indi

ana

Uni

vers

ityB

loom

ingt

on, I

N 4

7405

Dr.

Dan

iel R

eisb

erg

Ree

d C

olle

geD

epar

tmen

t of

Psyc

holo

gyPo

rtla

nd, O

R 9

7202

Dr.

Bri

an R

eise

rIn

stitu

te f

or th

e L

earn

ing

Scie

nces

Nor

thw

este

rn U

nive

rsity

1890

Map

le A

venu

eE

vans

ton,

IL

602

01-3

142

Dr.

Lau

ren

Res

nick

Lea

rnin

g R

& D

Cen

ter

Uni

vers

ity o

f Pi

ttsbu

rgh

3939

O'H

ara

Stre

etPi

ttsbu

rgh,

PA

152

13

Dr.

Gilb

ert R

icar

dM

ail S

top

K01

-14

Gru

mm

an A

ircr

aft S

yste

ms

Bet

hpag

e, N

Y 1

1714

Mr.

W. A

. Riz

zoH

ead,

Hum

an F

acto

rs D

ivis

ion

Nav

al T

rain

ing

Syst

ems

Cen

ter

Cod

e 26

1235

0 R

esea

rch

Park

way

Orl

ando

, FL

328

26-3

224

Dr.

Lin

da G

. Rob

erts

Scie

nce,

Edu

catio

n, a

ndT

rans

port

atio

n Pr

ogra

mO

ffic

e of

Tec

hnol

ogy

Ass

essm

ent

Con

gres

s of

the

Uni

ted

Stat

esW

ashi

ngto

n, D

C 2

0510

5 i

Dr.

Sal

im R

ouko

sIB

M C

orpo

ratio

nT

. J. W

atso

n R

esea

rch

Cen

ter

PO B

ox 2

18Y

orkt

own

Hei

ghts

, NY

105

98

Mr.

Lou

is R

ouss

osU

nive

rsity

of

Illin

ois

Dep

artm

ent o

f St

atis

tics

101

Illin

i Hal

l72

5 So

uth

Wri

ghi S

t.C

ham

paig

n, I

L 6

1820

Dr.

Edu

ardo

Sal

asH

uman

Fac

tors

Div

isio

n (C

ode

262)

1235

0 R

esea

rch

Park

way

Nav

al T

rain

ing

Syst

ems

Cen

ter

Orl

ando

, FL

328

26-3

224

Dr.

Fum

iko

Sam

ejim

aD

epar

tmen

t of

Psyc

holo

gyU

nive

rsity

of

Ten

ness

ee31

013

Aus

tin P

eay

Bld

g.K

noxv

ille,

TN

379

66-0

900

Mr.

Dre

w S

ands

NPR

DC

Cod

e 62

San

Die

go, C

A 9

2152

-680

0

Cap

itan

Jesu

s B

erna

l San

tos

Min

iste

rio

de D

efen

sa.

Uni

dad

de P

sico

logi

a (S

EG

EN

TE

)po

de

la C

aste

llana

, 109

2807

1, M

adri

dE

SPA

NA

Dr.

Dan

iel L

. Sch

acte

rD

epar

tmen

t of

Psyc

holo

gyH

arva

rd U

nive

rsity

Cam

brid

ge, M

A 0

2138

Dr.

Wal

ter

Schn

eide

rL

earn

ing

R&

D C

ente

rU

nive

rsity

of

Pitts

burg

h39

39 O

'Har

a St

reet

Pitts

burg

h, P

A 1

5260

Dis

trib

utio

n L

ist

Dr.

Mar

y Sc

hrat

z41

00 P

arks

ide

Car

lsba

d, C

A 9

2008

Dr.

Myr

na F

. Sch

war

tzD

irec

tor

Neu

rops

ycho

logy

Res

earc

h L

abM

oss

Reh

abili

tatio

n H

ospi

tal

1200

Wes

t Tab

or R

oad

Phila

delp

hia,

PA

191

41

Dr.

Rob

ert J

. Sei

del

US

Arm

y R

esea

rch

Inst

itute

5001

Eis

enho

wer

Ave

.A

lexa

ndri

a, V

A 2

2133

Dr.

Ter

renc

e J.

Sej

now

ski

Prof

esso

rT

he S

alk

Inst

itute

P. 0

. Box

858

00Sa

n D

iego

, CA

921

38-9

216

Dr.

Mic

hael

G. S

haft

oN

ASA

Am

es R

esea

rch

Ctr

.M

ail S

top

262-

1M

offe

tt Fi

eld,

CA

940

35-1

000

Dr.

Val

erie

L. S

halin

Dep

artm

ent o

f In

d. E

ngin

eeri

ngSt

ate

Uni

vers

ity o

f N

ew Y

ork

342

Law

renc

e D

. Bel

l Hal

lB

uffa

lo, N

Y 1

4260

Mr.

Ric

hard

J. S

have

lson

Gra

duat

e Sc

hool

of

Edu

catio

nU

nive

rsity

of

Cal

ifor

nia

Sant

a B

arba

ra, C

A 9

3106

Dr.

Tra

cy S

hors

Dep

t. of

Psy

chol

ogy

Prin

ceto

n U

niv.

Gre

en H

all

Prin

ceto

n, N

J 08

544

Dr.

Ran

dall

Shum

aker

Nav

al R

esea

rch

Lab

orat

ory

Cod

e 55

0045

55 O

verl

ook

Ave

nue,

S.W

.W

ashi

ngto

n, D

C 2

0375

-500

0

Dr.

Edw

ard

Silv

erL

RD

CU

nive

rsity

of

Pitts

burg

h39

39 O

'Har

a St

reet

Pitts

burg

h, P

A 1

5260

Dr.

Zita

M. S

imut

isD

irec

tor,

Man

pow

er &

Per

sonn

elR

esea

rch

Lab

orat

ory

US

Arm

y R

esea

rch

Inst

itute

5001

Eis

enho

wer

Ave

nue

Ale

xand

ria,

VA

223

33-5

600

Dr.

Jer

ome

E. S

inge

rD

epar

tmen

t of

Med

ical

Psy

chol

ogy

Uni

form

ed S

ervi

ces

Uni

v. o

f th

eH

ealth

Sci

ence

s43

01 J

ones

Bri

dge

Roa

dB

ethe

sda,

MD

208

14-4

799

Dr.

Jan

Sin

nott

Dep

artm

ent o

f C

ompu

ter

Scie

nce

Tow

son

Stat

e U

nive

rsity

Tow

son,

MD

212

04

Dr.

Der

ek S

leem

anC

ompu

ting

Scie

nce

Dep

artm

ent

The

Uni

vers

ityA

berd

een

AB

9 2F

XSc

otla

ndU

NIT

ED

KIN

GD

OM

Dr.

Rob

ert S

mill

ieN

aval

Oce

an S

yste

ms

Cen

ter

Cod

e 44

3Sa

n D

iego

, CA

921

52-5

000

Dis

trib

utio

n L

ist

Dis

trib

utio

n L

ist

Dr.

Ric

hard

E. S

now

Dr.

Sha

ron

Tka

czD

r. D

ougl

as W

etze

lD

r. R

ober

t A. W

ishe

rSc

hool

of

Edu

catio

nC

AE

-Lin

k C

orpo

ratio

nC

ode

15U

S. A

rmy

Inst

itute

for

the

Stan

ford

Uni

vers

ity20

9 M

adis

on S

tree

tN

avy

Pers

onne

l R&

D C

ente

rB

ehav

iora

l and

Soc

ial S

cien

ces

Stan

ford

, CA

943

05A

lexa

ndri

a, V

A 2

2314

San

Die

go, C

A 9

2152

-680

050

01 E

isen

how

er A

venu

eA

lexa

ndri

a, V

A 2

2333

-560

0D

r. J

ames

J. S

tasz

ewsk

iD

r. D

ougl

as T

owne

Dr.

Bar

bara

Whi

teD

ept.

of P

sych

olog

yB

ehav

iora

l Tec

hnol

ogy

Lab

sSc

hool

of

Edu

catio

nD

r. M

erlin

C. W

ittro

ckU

nive

rsity

of

Sout

h C

arol

ina

Uni

vers

ity o

f So

uthe

rn C

alif

orni

aT

olm

an H

all,

EM

STG

radu

ate

Scho

ol o

f E

duca

tion

Col

umbi

a, S

C 2

9210

1228

Spr

ing

Stre

etU

nive

rsity

of

Cal

ifor

nia

Uni

v. o

f C

alif

., L

os A

ngel

esSt

. Hel

ena,

CA

945

74B

erke

ley,

CA

947

20L

os A

ngel

es, C

A 9

0024

Dr.

Bru

ce D

. Ste

inbe

rgC

urry

Col

lege

Dea

n, C

olle

ge o

f B

ehav

iora

lA

lexa

ndra

K. W

igdo

rD

r. K

enta

ro Y

amar

noto

Milt

on, M

A 0

2186

and

Soci

al S

cien

ces

NR

C M

H-1

7603

-0T

Dr.

Fre

deri

ck S

tein

heis

erU

nive

rsity

of

Mar

ylan

d,B

altim

ore

Cou

nty

2101

Con

stitu

tion

Ave

.W

ashi

ngto

n, D

C 2

0418

Edu

catio

nal T

estin

g Se

rvic

eR

osed

ale

Roa

dC

IA-O

RD

Bal

timor

e, M

D 2

1228

Prin

ceto

n, N

J 08

541

Am

es B

uild

ing

Dr.

Dav

id W

iley

Was

hing

ton,

DC

205

05C

hair

, Dep

artm

ent o

f Ps

ycho

logy

Scho

ol o

f E

duca

tion

Dr.

Mas

oud

Yaz

dani

Dr.

Ted

Ste

inke

Uni

vers

ity o

f M

aryl

and,

Bal

timor

e C

ount

yan

d So

cial

Pol

icy

Nor

thw

este

rn U

nive

rsity

Dep

t. of

Com

pute

r Sc

ienc

eU

nive

rsity

of

Exe

ter

Dep

t. of

Geo

grap

hyB

altim

ore,

MD

212

28E

vans

ton,

IL

602

08Pr

ince

of

Wal

es R

oad

Uni

vers

ity o

f So

uth

Car

olin

aE

xete

r E

X44

PTC

olum

bia,

SC

292

08D

r. K

urt V

anL

ehn

Dr.

Dav

id C

. Wilk

ins

EN

GL

AN

DL

earn

ing

Res

earc

hU

nive

rsity

of

Illin

ois

Dr.

Kur

t Ste

uck

& D

evel

opm

ent C

tr.

Dep

artm

ent o

f C

ompu

ter

Scie

nce

Fran

k R

. Yek

,, vi

chA

L/H

RT

IU

nive

rsity

of

Pitts

burg

h40

5 N

orth

Mat

hew

s A

venu

eD

ept.

of E

duca

tion

Bro

oks

AFB

3939

O'H

ara

Stre

etU

rban

a, I

L 6

1801

Cat

holic

Uni

vers

itySa

n A

nton

io, T

X 7

8235

-560

1Pi

ttsbu

rgh,

PA

152

60W

ashi

ngto

n, D

C 2

0064

Dr.

Bru

ce W

illia

ms

Dr.

Mic

hael

J. T

arr

Dr.

Fra

nk L

. Vic

ino

Dep

artm

ent o

f E

duca

tiona

lD

r. J

osep

h L

. You

ngD

ept.

of P

sych

olog

yN

avy

Pers

onne

l R&

D C

ente

rPs

ycho

logy

Nat

iona

l Sci

ence

Fou

ndat

ion

Yal

e U

nive

rsity

San

Die

go, C

A 9

2152

-680

0U

nive

rsity

of

Illin

ois

1800

G S

tree

t, N

.W.,

Roo

m 3

20PO

Box

11A

, Yal

e St

atio

nU

rban

a, I

L 6

1801

Was

hing

ton,

DC

205

50N

ew H

aven

, CT

065

20D

r. J

erry

Vog

tD

epar

tmen

t of

Psyc

holo

gyD

r. K

ent E

. Will

iam

sM

r. A

ntho

ny R

. Zar

aD

r. K

ikum

i Tat

suok

aSt

. Nor

bert

Col

lege

Vir

gini

a Po

lyte

chni

c In

stitu

teN

atio

nal C

ounc

il of

Sta

teE

duca

tiona

l Tes

ting

Serv

ice

De

Pere

, WI

5411

5-20

99an

d St

ate

Uni

vers

ityB

oard

s of

Nur

sing

, Inc

.M

ail S

top

03-T

Dep

artm

ent o

f In

dust

rial

and

676

N. S

t. C

lair

, Sui

te 5

50Pr

ince

ton,

NJ

0854

1D

r. J

acqu

es V

onec

heSy

stem

s E

ngin

eeri

ng/M

SLC

hica

go, I

L 6

0611

Uni

vers

ity o

f G

enev

a19

00 K

raft

Dri

veD

r. G

ary

Tho

mas

son

Dep

artm

ent o

f Ps

ycho

logy

Bla

cksb

urg,

VA

240

60Pr

of. G

erar

d de

Zee

uwD

efen

se M

anpo

wer

Dat

a C

ente

rG

enev

aC

ente

r fo

r In

nova

tion

99 P

acif

ic S

tree

tSW

ITZ

ER

LA

ND

120

4D

r. M

ark

Wils

onan

d C

oope

rativ

e T

echn

olog

ySu

ite 1

55A

Gra

duat

e Sc

hool

of

Edu

catio

nG

rote

Bic

kers

stra

at 7

2M

onte

rey,

CA

939

40U

nive

rsity

of

Cal

ifor

nia,

Ber

kele

y10

13 K

S A

mst

erda

mB

erke

ley,

CA

947

20T

he N

ET

HE

RL

AN

DS

54

DOCUMENT RESUME ED 390 912 TM 024 257 …ED 390 912 AUTHOR TITLE INSTITUTION SPONS AGENCY PUB DATE...

Documents

Transcript of DOCUMENT RESUME ED 390 912 TM 024 257 …ED 390 912 AUTHOR TITLE INSTITUTION SPONS AGENCY PUB DATE...