Direct Comparison of Intuitive, Quasi-Rational and Analytical Cognition; Hammond.pdf

7/27/2019 Direct Comparison of Intuitive, Quasi-Rational and Analytical Cognition; Hammond.pdf

1/83

t 1xv ~ ~ i ~

04 -

'rI

DTI71..

4.. .~. 07 2 i3VF. A. oi. - -


2/83

,'I

DIRECT COMPARISON OF INTUITIVE,QUASI-RATIONAL AND ANALYTICAL COGNITION

Kenneth Hammond, Robert M. Hamm,Janet Grassia and Tamra Pearson

Center fo r Research on Judgment and PolicyUniversity of Colorado at Boulder

Report No. 248 ,- I.June 1983 JUL 1 2 1983

This research was supported by the Engineering Psychology Programs, Office ofNaval Research, Contract N00014-77-C-0336, Work Unit Number NR 197-038 and by K BRSG Grant #RR07013-14 awarded by the Biomedical Research Support Program, "Division of Research Resources, NIH. Center for Research on Judgment andPolicy, Institute of Behavioral Science, University of Colorado. Reproductionin whole or in part is permitted for any purpose of the United States


3/83

SECURITY CLASSIFICATION OF THIS PAGE (When Data Entered)REPORT DOCUMENTATION PAGE READ INSTRUCTIONSBEFORE COMPLETING FORM

I REPORT NUMBER j2. GOVT ACCESSION NO. 3. RPECIPIENT'S CATALOG NUMBER-CRJP 248 TP P OFRPR LR OEE

4 TITLE (and Subtitle) S TVPE OF REPORT & PERIOD COVEREDDirect Comparison of Intuitive, TechnicalQuasi-rational and Analytical Cognition 6. PERFORMING 040, REPORT NUMBER

7. AU THOR(s) S. CONTRACT OR GRANT NUMNER(I)Kenneth R. Hammond, Robert M. Hamm, N00014-77-C-0336Janet Grassia and Tamra Pearson

9. PERFORMING ORGAWiZATION NAME AND ADDRESS 10 . PROGRAM ELEMENT. PROJECT, TASKCenter for Research on Judgment an d Policy AREA&WORK UNIT NUMBERSInstitute of Behavioral ScienceUniversity of Colorado, Boulder, CO 8030911 . CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATEOffice of Naval Research June 1983800 North Quincy Street 13 . NUMBER OF PAGESArlington, VA 22217 71414. MONITORING AGENCY NAME & ADDRESS(II different from Controlling Office) 15. SECURITY CLASS. (of thia reporti)

Unclassified15q. E CL ASSI FI CATI N/Dow N GnAOINGSCHEDULE

16. DISTRIBUTION STATEMEN ' (o f this Report)

* Approved for public release; Distribution unlimited.

17. DISTfIBUTION STATEMENT (of the ob,,tract entered In Block 2-1, If different from Report)

IS.UPPLEMENTARY NOTES

19. KEY WORDS (Continue on reveree aide It necessary an d Identify by block rumber)achievement, cognitive control, confidence, consistency, environmentalmodels, experts, intuitive/analytical, Lens Model, cognitive continuumtheory, social judgment theory, task characteristics, transportation,ko safety

20. ABSTRACT (Continui on reverse side if necessary and Identify by block number)The relative efficacy of intuitive an d analytical cognition inanalytically competent persons was directly compared. More subjectsperformed best in the intuitive mode when inconsistency was removed fromtheir judgments, an indication that the subjects possessed implicitknowledge that they did not utilize in the analytical mode. More subjectsmade larger errors in the analytical mode than in the intuitive mode.Subjects' confidence was generally inappropriately placed.

DD I , 1473 EDITION OF I NOV 65 IS OBSOLETE


4/83

Direct ComparisonHammond, Ham, Grassia, and PearsonAck nowledgments

In planning our research we received indispensable assistance fromMatthew Reay, Barbara Chokol, Richard Cutler, Robert Whitson, and JosephJuhasz. Alexander Ariniello, Mac Best, Daniel Cronin, Betty Davie, DarylFleming, John McEahern, Clark Misner, William Pollard, Jerold Simpson, andMiro Supitar gave us valuable advice and served on our expert panels. John H.Allen, John Bright, Charles R. Keim, and Marilyn Kuntemeyer were pilotsubjects.

We thank Harvey Atchison, Robert Clevenger, John Dolan, Edward Haase, Don(0 Heiderstadt, Steven Holt, William Tucker, Kells Waggoner, and Theodore

Zimmerman. These administrators in private firms and government organizationsin the Denver metropolitan area allowed members of their staffs to participateas subject5 in our research.

To the twenty-one subjects whose judgments of safety are described inthis report, we are especially grateful: David Baskett, William Bohnhoff,Warren Brown, Vincent Cerasi, Denis E. Donnelly, Richard W. Hain, AndrewHollar, David Jessup, Bruce Johnson, Robert L. Kenny, David Leahy, RonaldMarschel, Phillip McCabe, Keith G. Meyer, Eugene G. Muhicn, Joseph Plizga,Roland C. Rautenstraus, Steven D. Rudy, Arnold J. Ullevig, Merle Van Den Bos,and Phillip Weisbach. Without their generous contribution of time and hardwork our research would not have been possible.

Finally, we deeply appreciate the skill and attention to detail whichDoreen Victor and Mary Luhring have put forth in editing this paper and themany other documents necessary to our research.


5/83

* Direct Comparison Page 1Hammond, Hamm, (rassia, and Pearson

* ' ~TAIBl

DIRECT COMPARISON OF INTUITIVE, QUASI-RATIONAL . ..AND ANALYTICAL COGNITION

1 ,iiAL LComparison of the efficacy of intuitive and analytical cognifffi6Thaf'ben

a major topic in judgment and decision research since it s inception (for areview of approaches to this topic, see Hammond, McClelland & Mumpower, 1980).Although intuitive cognition is largely believed to be inaccurate,systematically biased, and to produce judgments in which persons areoverconfident, (for reviews see Slovic, Fischhoff & Lichtenstein, 1977;Einhorn & Hogarth, 1981), Hastie in his review (1983) concludes that: "noneof the applications [of probability theory] to social judgment provide

compelling arguments that human reasoning deviates from normative modelprescriptions" (p. 511). Our premise is that both conclusions are based onresearch that is restricted in the following ways:

1. Comparisons are indirect.Th e most widely used research method compares the intuitively derived

judgments of subjects with analytically derived answers generated byformal models such as Bayes' Theorem and multiple regression equations(see, for example, Kahneman, Slovic, & Tversky, 1982; Hammond, Stewart,

* Brehmer, & Steinmann, 1975). Informative as these comparisons may be,they do not directly compare the achievement of intuitive and analyticalcognition undertakcn by the same person. Direct comparisons are needed in


6/83

", ., j . ~ -r r . - . r~ r r

Direct Comparison Page 2Hammond, Hamm, Grassia, and Pearsonorder to know which mode of human crognition is superior. (For a generalexposition regarding these comparisons see Tweney, Doherty & Mynert,1981.)2. Research subjects lack analytical competence.

Research subjects--typically college sophomores--generally lackknowledge of an y analytical principles that would enable them to erigage inanalytical cognition. Moreover, they are seldom given the time oropportunity to do so . Since the subject's analytical competence is notexamined in parallel with his/her intuitive competence, the substitutionof formal models to represent analytical cognition constitutes a special

- comparison. That is, intuitive cognition is set in competition with a"form of analytical cognition refined by centuries of study by thousands ofindividuals over their lifetimes. Although such comparisons areundeniably important, they do not exhaust our range of interest in

- cognition; they do not inform us about what happens when the same personengages in different forms of cognitive activity. They cannot, forexample, inform us about the possible disadvantages as well as advantagesof engaging in analytical cognition.3. Intuition and analysis remain obscure concepts.

Researchers in cognitior almost never explain what they mean byintuition, although they take great pains to differentiate precisely amongformal, analytical models of cognition. A3 a result it is customary toexplain that intuition is what analysis is not. Brooks (1978), forSexample, compares "analytic and non-analytic concept formation." Kahneman

- and Tversky (1982) indicate that "a judgment is called intuitive if it is


7/83

Direct Comparison Page 3Hammond, Hamm, Grassia, and Pearson

reached by an informal and unstructured mode of reasoning, without the use40 of analytical methods or deliberate calculations" (p. 494). Lven

philosophers (e.g., Cohen, 1981) who criticize psychological researchregarding intuitive judgments of probability fail to say what they mean by

O this term. As a result, the task properties that induce either form ofcognition are not specified, thus making their comparison difficult, ifnot impossible.*4. Methodological and substantive contion are not differentiated.

Successful cognitive efforts require substantive as well asmethodological competence; neither alone can insure empirical success(cf., Hammond, 1966, pp . 68-75; Wason & Johnson-Laird, 1972; Adelmen,1981). Topical knowledge can be as important as the knowledge ofmethodological principles. These two aspects of cognition are generallystudied separately, however. Judgment and decision researchers contrasttheir subjects' intuitive methodological cognitior (efforts to combineinformation) with analytically derived normative methodological rules forcombining information (e.g., Bayes' Theorem), irrespective of subjects'topical knowledge (see Einhorn & Hogarth, 1981; Slovic et al., 1977).Researchers who study proolem-solving, on the other hand, generallycontrast their subjects' manipulation of the substantive materials of thetask with scientific truths or experts' behavior (see, for example,Larkin, McDermott, Simon & Simon, 1980; also Duda & Shortliffe, 1983)without regard for subjects' methodological competence. If the efficacyof intuitive and analytical cognition are to be directly compared,however, these two aspects of cognition need to be examined together, forthey can be compensatory in many task circumstances. (An explicit effortto study both can be seen in Fox, 1980.)


8/83


5. Evaluations of intuitive methodological cognition are norm-conti gent.

Considerable uncertainty exists about which normative rules should beused to evaluate intuitive methodological cognition. The sharplydivergent views of philosophers on this topic (see, fo r example, Kyburg &Smokler, 1980, Kyburg, 1974, and Levi, 1980) are reflected in theresponses to Cohen's (1981) criticism of judgment and decision researchthat uses the standard probability calculus to evaluate intuitivemethodological cognition. The number and variety of normative inductiverules cited in these responses as the proper rule to follow are clearevidence that both frequentists an d subjectivists have failed to solve the

* -fundamental problem of induction. Until they succeed, the researchpsychologist who attempts an empirical evaluation of the extent to whichintuitive cognition conforms to normative rules will inevitably produceresults contingent upon the normative rules s/he prefers--as those whoprefer different rules will be quick to point out. Einhorn and Hogarth(1981) cite other difficulties with normative rules in their review whenthey note that choice of an optimal model is "conditional on certainenvironmental assumptions and a specific time horizon" (p. 55) and"conclude their discussion by observing that "To consider humaji judgment assuboptimal without discussion of the limitations of optimal models is"naive" (p. 56).

0P Evaluations of substantive analytical cognition are less vulnerableto norm-contingent restrictiqns, however, because a subject's productionof a substantive law or rule can be evaluated with respect to well -agreed

- upon empirical events (cf. Scribner, 1977, who distinguishes between"theoretical" and "empirical" standards for evaluating logical processes).Therefore, empirical achievement is a criterion for cognitive activity

- - -T - 7


9/83

Direct Comparison Page 5Hammond,, Hamm, Grassia, and Pearson

that deserves as much or more interest as conformity with one, among many,normative rules. (See Hammond et al., 1980, on "logical vs. empiricaloptimality," p. 215ff; see also Einhorn & Hogarth, 1982, on "truth vs.accuracy".)

6. Evaluations of intuitive methodological cognition are task contingent.

Current conclusions about intuitive competence in human beings arelargely restricted to the performance of subjects working withmethodological problems that have been constructed to be readily treatedby the methods of the subjective interpretation of the standardprobability calculus. But, as noted above, cognitive problems involvesubstantive competence as well as methodological competence. Therefore,comparison of the efficacy of intuition and analysis requires examinationin tasks that offer challenges to empirical accuracy as well as intuitivemethodological competence.

* Examination of performance in substantive tasks representative of aperson's intellectual environment should be at least as instructive as anexamination of performance in task conditions specifically constructed toconform to normative methodological rules. Johnson-Laird a.d Wason, forexample, note in their 1977 "Postscript": "by attempting to relatethe ... task more closely to the subjects' experience, performance wasdramatically improved" (1977, p. 151). Einhorn and Hogarth (1981)conclude: "It is essential to emphasize that the cognitive approach hasbeen concerned primarily with how tasks are represented. Th e is.,ue of wh ytasks are represented in particular ways has not yet been addressed" (p.57). Indeed, the general absence of the description and classification ofcognitive tasks has long been emphasized as a serious shortcoming in


10/83

Direct Comparison Page 6Hammond, Hamm, Grassia, an d Pearson& Lichtenstein, 1971; Linhorn & Hogarth, 1981) an d continues to be(Hastie, 1983), but without tangible result.In sum, in addition to making indirect comparisons it is necessary to

directly compare an analytically competent person's use of intuitive andanalytical cognition, to permit the subject to employ either or both inrelation to substantive as well as methodological cognition, an d to evaluatehis or her achievement empirically. In the present study, therefore, we dothe following:

1. directly compare intuitive and analytical cognition in the sameperson;

2. use as subjects analytically competent persons so that both intuitiveand analytical cognition can be brought to bear on the same task;

3. specify the properties of intuitive and analytical cognition and theproperties of tasks that evoke each;

4. include substantive as well as methodological aspects of cognition;5. contrast the efficacy of intuition and analysis in terms Jf empirical

achievement; and6. use problems representative of the subject's intellectual environment.

- - - ---- - -- .- .-,--.--- . - - - - .


11/83

Direct Comparison Page 7Hainond, Hamm, Grassia, and PearsonTheoretical Bac__round

A Cognitive Continuum vs. a Cognitive Dichotomy_

Our basic premise is that cognitive activity is not a dichotomy ofintuition and analysis but rather a continuum marked by intuition at one poleand analysis at the other. Unlike the traditional dichotomous premise, acognitive continuum permits a quasi-rational compromise between intuitive and

L analytical cognition; thus a form of cognition more common than either pureintuition or analysis may be described. (For previous use of the concepts ofcognitive continuum and cognitive compromise see Brunswik, 1952, 1956;Hammond, 1955; Hammond & Brehmer, 1973; Hamnmond et al., 1975; Hammond etal., 1980; Hammond, Note 1; see also Anderson, Deane, Hammond, McClelland &Shanteau, 1981.)

Modes of Cognition

Intuition and analysis can be distinguished by the relative degree of:(a) cognitive control (in intuitioi, low; in analysis, high); (b) rate ofdata processing (in intuition, rapid, i.e., as brief as microseconds; inanalysis, slow, i.e., as long as hours); (c) conscious awareness of process(in intuition, low; in analysis, high); (d) ype of organizing principl_ (inintuition, a weighted average; in analysis, other, task-specific principles);(e) type of error (in intuition, normally distributed; in analysis, few, butlarge errors); (f) type of confidence (in intuition, confidence in answer butnot method; in analysis, confidence in method, not answer). (For furtherdistinctions, see Hammond, Note 1.) The compromise form of cognition,quasi-rationality or "common sense," includes properties from both types ofcognition and, therefore, is described in terms of the number and nature ofthe cognitive properties it includes from both.

; p1. - - -


12/83

iirect Comparison Page 8Hammond, Hamm, Grassia, and Pearson

Task Properties That Induce Different Modes of Cognition

If, as in the present study, subjects are not provided with feeaback, thetask properties that differentially induce intuition and analysis include:(a) number of cues available (in intuition, large [> 5]; in analysis, small);(b) the order in which cues are displayed (in intuition, simultaneous; in

, - analysis, sequential)- (c) the type of cue measurement required (inintuition, perceptual; in analysis, objective, as with instruments); (d) cuedistribution characteristics (in intuition, continuous, highly variable,normally distributed; in analysis dichotomous, valued in terms of specificnumbers, distributions unknown); and (e) redundancy among cues (in intuition,high; in analysis, low). (See Hamnw)nd, Note 1, for further elaboration.)

*l Quasi-rationality is induced to the extent that tasks contain properties fromboth types of polar task conditions. We do not claim that al l of theseproperties must be present in order to locate a task at either pole of thecontinuum, nor do we know their relative importance or their interactive

* effects that produce quasi rationality. We assert only that specification ofthese task properties is a useful guide fo r direct comparison of the relativeefficacy of different modes of cognition.

IObjectives

* _-Four aspects of the cognitive continuum theory are examined in thell context of the six conditions indicated above. Because of the long-standinginterest in the relative efficacy of modes of cognition we first examinedifferences in the empirical accuracy of highway engineers' judgments ofsafety in the intuitive, quasi-rational and analytical modes. Highwayengineers were chosen as subjects because of their frequent professional useof all three modes of cognition. Judgments of highway safety were chosen


13/83


because, as we show below, the properties of this task should induce quasirationality, thus illustrating the importance of the middle range of thecognitive continuum. (An intuition-inducing task--judging highwayaesthetics--and an analysis-inducing task--judging highway capacity--were also*empicyed, but space prevents their discussion here.)

Second, because of the important theoretical role of cognitivecontrol--predicted to be low in intuition, middle-level in quasi rationality,and high in analysis--we examine differences in cognitive control in each ofthe three modes. More specifically, the relative contributions of knowlengeand cognitive control to accuracy are compared. Tvii:a two aspects ofcognition, identified by Hammond and Summers (1972), have been found to beempirically significant in numerous studies of multiple-cue probabilitylearning (see, e.g., Brehmer, 1979), of interpersonal conflict andinterpersonal learning (see Brehmer, 1972, 1976; Holzworth, in press;Brehmer & Hammond, 1977 fo r a review), of the differential effects of variouspsychoactive drugs on interpersonal learning and interpersonal conflict amongpsychotic patients (Gillis, 1975, 1978), and in clinical judgment (e.g.,Fisch, Joyce, Hammond & O'Reilly, 1982; Kirwan, Chaput de Saintonge, Joyce, &Currey, in press). In this study knowledge and cognitive conti-ol are examinedto determine whether they provide different contributions to achievement underthe three modes of cognition within the same person.

One of the advantages of the direct comparison of human intuitive andanalytical cognition is that the errors made in the analytical miiode ofcognition can be observed and compared with the errors of intuition. Thiscomparison cannot be made when intuitive cognition is compared only with aformal niodel. For in that case the answer provided by the formal modelexhausts the concept of truth; by definition, errors do not exist. Therefore


14/83

Direct Comparison Page 10Hammond, Hamm, Grassia, and Pearsonour third objective is to compare the errors produced by analytical as well asintuitive and quasi-rational connition.

Brunswik has already argued and demonstrated (195b, pp. 89-93) thatintuitive cognition produces errors normally distributed around the correctanswer because "intuitive perception must integrate many avenues of approach,or cues...none of which is foolproof or fully ecologically valid" (p. 92).Because analytical cognition proceeds in the opposite fashion, it producesmany exactly correct answers, but its errors are likely to be extreme. Wetherefore ask whether intuitive cognition produces a normal distribution oferrors centered on the correct answer; whether analytical cognition produces 40precisely correct answers together with highly incorrect answers; and whetherquasi-rationality produces a distribution of errors lying between thosegenerated by the polar modes of cognition. 4

Fourth, the relation between confidence and performance is examined todiscover whether confidence matches performance (Oskamp, 1965; Lichtenstein,Fischhoff & Phillips, 1982): is an engineer most confident in that cognitivemode in which he performed best and least confident in the mode under which heperformed most poorly? The theory outlined above suggests an additionalhypothesis. Since the method of intuitive cognition produces rapid,nonretraceable answers and since these answers are based on multiple "avenuesof approach or cues," subjects should be less confident in the intuitivemethod than in intuitive answers. The opposite should be true in theanalytical mode, in which the subject's attention is fociised on theorganization of cues.


15/83

Direct Comparison Page 11Hammond, Hamm, Grassia, and PearsonMethod

Subjects

Twenty-one male highway engineers, 30-70 years of age, served as researchsubjects. Since engineeers are professionally trained to cope with problemsthat have substantive analytical components, the intuitive and analyticalcognitive efforts of the same subject may be directly compared. Theengineers' task was to evaluate the safety of highways, a complex problemrepresentative of those encountered in their work.Independent Variables

In the intuition-inducing condition film strips of one- to three-milesegments of forty two-lane rural Colorado highways were presented. Engineersjudged the safety of each segment solely on the basis of the visual materialirn the film strips (see Figure 1). This form of presentation meets theconditions for inducing intuitive cognition by requiring the engineers toobserve a large number of cues contemporaneously displayed, and to measurethem by unaided visual perception. The values of the cues are generallycontinuous and normally distributed; the cues are frequently r-dundant. Theengineers were neither asked nor given the opportunity to organize the taskmaterials explicitly.


16/83

- Direct Comparison Page 12Hammond, Hamm, Grassia, and Pearson

.44

Figure 1. View of Highway #15 from Film Strip

In he condition designed to induce quasi rationality, the same forty

4,,.

0 highway segments were presented as bar-graph profiles which displayed valuesfor ten dimensions (see Figure 2). The bar-graph presentation meets thespecifications for inducing quasi-rational cognition by combining intuition-and analysis-inducing properties. On the one hand, the task remainsintuition-inducing because the number of cues is still large; they areredundant and contemporaneously displayed; and they have continuous, largely

S.. ; "I


17/83

Direct Comparison Page 13Hamnond, Hanoi, Grassia, and PearsonHIGHWAY JUDGMENT PROJECT !982

BAR GRAPH PRESENTATION. SAFETY JU[)GMENT TASK.

8 10 13Lane Width XXXXXXXXXXXXXXX

0 4 10Shoulder Width XXXXXXXXXXXXXXX

0 41 80Percent No Passing Zone XXXXXXXXXXXXXXXXXXXX

0 1 7Curves per Mile XXXXXX

0 47 75Grade XXXXXXXXXXXXXXXXXXXXXXX

0 470 10000Traffic Volume XXX

0 21 35Traff ic Mix XXXXXXXXXXXxxxxxxxxxxX

0 .7 4Intersections pe r Mile XXXXXXX

30 42 60Average Speed Limit XXXXXXXXXXXXXXX

0 4.4 18Obstacles pe r Mile XXXXXXXXX

Figure 2. Bar-graph Profile of Highway #27


18/83

* Direct Comparison Page 14Hammond, Hamu, Grassia, and Pearsonnormal distributions. The engineers were not given the opportunity toorganize the task materials explicitly and were not asked to explicate orjustify their judgments. On the other hand, the number of observable cues is

. reduced to ten. The cue values are expressed as numbers, and thus theengineers are not required to measure them perceptually. Th e bar-graphprofiles also permit easy cross-comparison of the values of various cues.

g In the analysis-inducing condition each engineer was requested toconstruct and justify a mathematical equation for predicting safety (seeAppendix A for examples). Th e subject was instructed to think; pencil, paperand hand calculators were made available; the engineer was told that acertain amount of time would be allowed for the completion of the formula, butthat the time period would be extended until he completed the task. Eachengineer was assigned to one of three subgroups within the analytical elcondition. In the "minimal guidance" subgroup (n = 12) engineers were allowedto work through the task with a minimum of instructions and requests from theresearchers. Six engineers in this group were encouraged to complete theirformulas in twenty to thirty minutes; the other six were give a target offorty-five minutes. In the "think aloud" subgroup (n = 6) engineers wererequested to think aloud as they constructed their equations so that their, 0analytical efforts could be monitored during the task. In the "maximalguidance" subgroup (n = 3) engineers constructed their equations according toa detailed written procedure (see Appendix B) designed to increase thelikelihood of a systematic approach to the problem. (Samples of engineer'sremarks regarding their cognitive activity in each mode are presented in

. Appendix C.)


19/83

SDirect Comparison Page 15Hammond, Hamm, Grassia, and PearsonThese three sub-conditions were used because they offered a variety of

* information. The minimal guidance condition revealed how engineers would goabout constructing a formula on their own. The "think aloud" conditionprovided an understanding of the engineers' process of constructing an

* equation which would not otherwise have been gained (see Appendix D). Themaximal guidance condition showed whether a structural approach was feasible.

It was not our purpose to determine the effects of the different LDsub-conditions. As would be expected, however, engineers in maximal guidanceand engineers who were given a 45-minute target in minimal guidance took moretime than the others. There were no significant differences in achievementamong the subgroups.Dependent Variables

The effect of the intuitive, quasi-rational, and analytical conditionswere examined with respect to (a) each engineer's degree of achievement inpredicting safety accurately, (b) the differential contributions of knowledgeand cognitive control to this achievement, (c) the relative frequency ofdifferent types of errors made in each of the three modes of cognition, and(d) the relation of confidence to performance in each mode.

Achievement was measured by the correlation between an engineer'sestimate of safety and the accident rate of each highway segment (thecriterion).

77

- aA~..A-A- A1

q-17


20/83

, Direct Comparison Page 16"Hammond, Hamm, Grassla, and Pearson

Knowledge and cognitive control were derived from the parameters of theLens Model Equation (Hanmnond et al., 1975):

ra =GR R +C[1 R /I R-where

ra achievement, the correlation between the engineer'sjudgments and criterion values

G = the correlation between judgments and criterion valuescorrected for attenuation due to less than perfectlinear predictability in each

R = environmental predictability (linear form)R = subject's predictability (linear form)C correlation between residuals from linear predictions

of criterion and residuals from linear predictions ofsubject's judgments.

In the absence of significant correlations between residuals (trivialvalues of C in the above equation) then ra = G Re Rs. Under these cond'itions"G epresents the engineer's knowledge because it indicates what the subject'sachievement would have been if he had executed his judgmenc policy withperfect cognitive control (i.e., R. = 1.00) and if the environmental taskcriterion were perfectly predictable from the cues (i.e., Re = 1.00).

Cognitive control is appropriately measured by Rk in the equation sincethere was little evidence in this study of lack of fit of the linear model.(See Hammond et al., 1975, for a detailed discussion of the distinctionbetween consistency and cognitive control.)

- - ---


21/83


- ifferentiel errors produced by each engineer are evaluated by (a)n examination of the distribution of deviations of judgments around the correct

answer, and (b) examination of mistakes made in the process of formulating theequation and evaluation of their consequences for accurate prediction.

Confidence in answers in the intuitive and quasi-rational modes ismeasured by the mean of the engineer's confidence, on a 1-10 scale, in each ofhis judgments. Confidence in answers in the analytical mode and confidence inmethod for al l three modes was measured by questions at the conclusion of thesession.

Procedure

Statistical Properties of the Task

The same se t of forty hiqhway segments was used in all conditions. Thecriterion for the accuracy of judgments is the accident rate, averaged over

9 seven years, for each highway segment. Accident rate is defined as the totalnumber of accidents (involving fatalities, injuries, or property damage only)divided by the number of vehicle miles traveled. Due to it s extremely highaccident rate, one highway was dropped from the analysis.

Highways were measured on ten dimensions, chosen for inclusion in the"study on the basis of discussions with highway safety experts who indicatedthe information they considered essential for evaluating the safet.y of a road(see Table 1 for list). Eight of these measures were available from highwaydepartment records; two measures (number of curves per mile and number ofobstacles per mile) had to be counted by the experimenters from visualinspection of film strips of each highway segment. The beta weights for eachdimension or cue in predicting accident rate are also presented in Table 1.


22/83


Visual examination of the scatter-plots of the relations between each cue andthe criterion indicated little if any nonlinear co-variation. This findingwa s supported by the results from calculation of the contribution of squaredterms and interactions to accident rates.

Table 1

Highway Characteristics (cues) Related to Accident Rate

Cues Beta weights

Lane Width .023Sthoulder Width -. 042Percent No Passing Zone -. 143Curves per Mile .152Grade .055Traffic Volume -. 198Traffic Mix (% of Trucks) -. 017Intersections per Mile .247Average Speed Limit -. 316Obstacles per Mile .478


23/83

Direct Compaarison Page 19Hammond, Hamm, Grassia, and PearsonAn optimally weighted linear multiple regression model of the task

indicates that Re = .b63, corrected for estimated shrinkage = .809.Application of equal weights and linear functions, each cue given the signthat appeared in the best fit equation, yields an Re of .769 (corrected for

0 shrinkage = .667). Th e intercorrelations among the ten cues and the criterionare presented in Table 2.

I Rating Scales

The judgment scale fo r each task condition was appropriate to thecognitive activity induced. An abstract rating scale from 1 (safe) to 10

(.p (unsafe) wa s used for the film strips to induce intuitive cognition. In thebar-graph presentation and in the task requiring the construction of aformula, a scale from 0 to 32 accidents per million vehicle miles traveled wasemployed because this specificity is compatible with calculation and thus withanalytical cognition. Transformations to a common scale were made forpurposes of data analysis and are described below.

TrialsAll engineers were presented with the tasks in the same order: first,

the film strips; second, the bar graphs; third, the materials for formulaconstruction. It was not appropriate to counterbalance the order ofpresentation because analytical work requiring use of certain cues in anexplicit fashion would have strongly influenced subsequent intuitivejudgments, whereas the reverse is not true (see Jones and Harris, 1982).

In the intuition-inducing mode, ten of the forty highways were showntwice; in the quasi rationality-inducing mode -ixteen highways were showntwice. These repetitions permitted calculation of repeated trials reliability

*for each engineer. t

- -V' -I. T-7,v T- -1,, - . -rr- r


24/83

Direct Comparison Paq 20Hammiond, Hamm, Grassia, and Pearson

IID

40N LAi 0i

CD r -0 0 CLO

00~ ~~A 0h4 LCnC~C3 L 'JO Ch a

(00

IL. M 0 OD0

UO %0 0 00 CA Mr4 UA

0 O 0~ CD '.YO OV) 0t 0t-C' 0. (nJ

U-0 (.0.*n LO L I- I~ I" In C)

-ja_U 0.JL0 LL.L L* LN- C5- V) 4 '.JC< LA N L)

4.) (DC


25/83

Direct Comparison Page 21Hammond, Ham, Grassia, and Pearson

Time

Response times were recorded in all three presentation modes, but areartifacts of the task conditions rather than descriptions of the subjects'

* behavior; that is, presentation of a film strip required more time thanpresentation of a bar graph. The mean response time in the intuitive mode was78 seconds and in the quasi-rational mode, 22 seconds.

0In the analytical mode the response time varied within the three

subgroups. The eighteen engineers in the think aloud and minimal guidancesubgroups were encouraged to complete their formulas in twenty to forty-fiveminutes; but seven of them took at least an hour at this task (maximum = 115minutes). No time constraints were imposed on the engineers in the maximalguidance condition; response times ranged from 138 to 250 minutes with a meanof 180 minutes.

RESULTS

Relative Efficacy of Three Modes of Cognition

Achievement

Individual differences in achievement. Achievement (ra) is measured byathe correlation between an engineer's judgments about the safety of each ofthe 39 highways and the accident rates of those highways. Data from repeatedjudgments in the intuitive and quasi-rational modes were not included. Forthe analytical condition, the engineer's formula was applied to each of the 39highways, and the answers thus produced were correlated with the accidentrates.


26/83

AL Direct Comparison Page 22Hammond, Hamm, Grassia, and Pearson

Wide individual differences occurred in all modes of cognition (idble 3).In the intuitive mode, the engineer's achievement correlations ranged from.071 to .636, with a median of .576. In the quasi-rational mode, ra rangedbetween .000 and .738, with a median of .516. In the analytical mode, ra sranged from -. 226 to .731, with a median of .467. Mean interjudge agreementin the intuitive mode is .58; in the quasi-rational mode, .47; and in theanalytical mode, .17. Th e correlation between engineers' achievement in theintuitive and quasi-rational modes is .344; between achievement in theintuitive and analytical modes, .457; and between achievement in thequasi-rational and analytical modes, .032 (Table 4).

Since the engineers' general policies for judging highway safety in the* intuitive and quasi-rational modes were all modeled by best-fit multiple

regression analysis, individual differences in achievement in these modescannot be attributed to the structure of this single model. In the analytical

", mode, however, judgment policies were not modeled but instead directlyexpressed by the engineers as formulas. Individual differences in achievement* in the analytical mode could have been caused, in part, by the wide variety in

S. the structure of those formulas (see Appendix A). It turned out, however,that the various structural features of the formulas are ,,ot related toachievement.

In the intuitive mode, the more experience the engineer had (measured byage, years of work, or years of education), the lower his cognitive control,R otherwise there were no significant relations between experience and thevarious measures of achievement, knowledge and consistency. The existence ofwide individual differences in achievement among the engineers in al l threecognitive modes thus supports the decision to study separately the performance

, ~ ~ ~ ~ ~ ~- 7-r . -- ~ -~, -. --. - --


27/83

Direct Comparison Page 23 .Hammond, Hamm, Grassia, and Pearson

CpC .jC) ) C) CD C)) C) C) C D CD 0D CDCI C) C) C) Cj) C0 C0 C) C) C) C) c:) C)

cyC) IC)0 O - c l 160) c t) 0) PC)L

c~f Ci LO 00j -40) L) M )0)al 0 00 I C)

Qu C7) " 0C) C) 0o 0 )0 C)0 0d C.* ) C)

4-)

C) C) C.Q C.) 'C) C) C) 10 CDA C) C-0 Cl CCD- O 0N CIJ C\J C-4 C-4 CD C C') C.0 C) CD ) C)C

O 0C) 0) to M4 qr r" a) %4 1.0 1.1) Li- e.J 1-4 LO %L)) 0. 0-44 M~ MV4 - ON 00) rLA to Ol 00~ CJ C r9

a4-00 C CDc) --- --- M_- 4w U)t

CL-~N 'J C. - (Y) C) (v) N-_ LA 1.0 M P-% C. ) U0.) ~~C:) 00 1. 0 qCO C) ~- ~- C:) LA -c t C.) LA) (V) C). tic qr- (V) c) C\J t-4 N- ko. 10 co LA C) MA %.0 Itd C) LA

.0H S.- -

0-) -- 1,- -;t 1.0 LA- qt MO M~ -6j rIt -A 1 t N -4* MI4J LO CO' LA4 r-= ON to N-I co C. co cn LA C_ N- CE C

0) 4.N C) 14 to) 1.0 'Ln P1.0 C) IN -) LA C-i cO

LA N- j cm.O WD1 W.- CO LA C O to C)NcoWU Cy M.N -- D- all w _ --(.IM-

0) LA -O -4 .- 4 W CO %04 w- () 10 C)i M. 10-l_ I'.0 C)'.. I. I--4 co FlA r- r-_4 L..~. Co .- 'C'. C) toJ-4 1 LA LA N- It C nC. L") CAr CIO

L.LiOMJ M N- j M'. q3" 100 C)0 CO 1.0 M) LcO -

a) ---- --- -__

Direct Comparison


28/83

Page 24Hammond, Hamm, Grassia and Pearson

0 C) C CD 0D C CS0 C) i 0 C0 0= C)4j~

-4-) -0 -9

Lt t -4-4 CO LOM Ln l, N-1 LO LO i00 r CO 0) N- CC)

) ro C CD C 0 0 0C

CC C) C) C=) C) CO CD C)

4-.)-M CLO M- V."

(A Cy Krt ) CY% 0A- O~l 01- 00 00- N- N- LA N-* 0.) 0 )

4-) L4-)P-4 '0 -4 LA M-- O*~ ~O Cl C O N- 0 i t "

4-) 0) 0C: -a C it iO V-4" 0\N.0 0 r- 0.D-% N- qt)t MOA '.'0U 0. A '.0 C -. LO rt U'.on Lu

E r-a 0o 4- LA '.0 0~) U-, 'o it rl-P'-4 0) -4 0" Cr3 CM (NJ N.

- 00ck C N 0Y) to I0 CI

(1) 4-.00 1-4 MO Cv) M4 11%

U) LA LA II) LOA-wU * . . . .0

6- 4 Cl Cj(\

(00Z o:

Direct Comparison Page 25


29/83

Hammond, Hamm, Grassia, and Pearson

CD,,,,

m "4- i

I-,--

0S.-

00o a) >

*U,Ecu

S.CU%

4-),

S.-a) __ __ ___

cuI5- Cl!J

V.

C"p

Cy



30/83


of each engineer (cf. Brunswik, 1952; Luce, 1959; Tversky, 1972; Shanteau,1975; Hammond et al., 198U, pp. 115-127; Hammond & Wascoe, 1980; Hammond,Note 1).

Achievement in three modes of cognition. In order to make directcomparisons of each subject's perfornance in each mode of cognition, theefficacy of each mode for each engineer was evaluated. Th e basic analysis ofdata thus consisted of rank ordering each engincer's performance across modes.Means are provided fo r general information but are not included in thestatistical tests.

A chi-square test of the hypothesis that one mode of cognition wassuperior to the others resulted in a failure to reject the null hypothesis(see Table 5). There were, however, only four engineers wnose achievement washighest in the intuitive mode. Comparison of the six orders of achievementdid result in rejection of the hypothesis of a chance distribution (p < .01).The data suggest that all nine engineers who performed best in he analytical

* nmode performed better in the intuitive than the quasi-rational mcde.

Components of achievement. As the Lens Model Equation indicates,achievement (ra) is a function of knowledge (G) and cogniti'.'e .ontrol (Rs).Since tw o subjects might have equal achievement for different reasons--thatis, their knowledge and their cognitive control over the application of Ztheirknowledge might vary in a compensatory manner--we determined the relative LIcontribution of these two components for each engineer in each cognitive mode.

, k& * A

Uirect Comparison Page 27


31/83


Table 5

Number of Engineers with Highest Achievement (ra) in Each Conditiona

Ist Order

I >Q>A 2I >A>Q 2

Q Q>I>A 6O > A > I 2

A >I>Q 9A > Q > I 0

21 21

X 2.00, p > .20 x2 = 15.86, D < .01

For the intuitive and quasi-rational data sets, the values of theparameters of the Lens Model Equation were calculated from the best fit linearmodels of the environment and of each engineer's judgments. For theanalytical data set, the best fit linear model was used fo r the environmentalsystem, but the engineer's judgment policy was represented by the formula heproduced. In some cases the formulas had nonlinear features. An additional



32/83

Hammond, Hamm, Grassia, and Pearsonanalysis was carried out using a linear best fit model of the answersgenerated form the engineer's formula; the results differed only slightlyfrom those produced by the first procedure and will not be reported here.

Knowledge. Table 6 shows that the knowledge component, G, for thirteenof the twenty-one engineers is higher in the intuitive mode than in thequasi-rational and analytical modes (X2 = 8.86; p < .02). Examination oforder also shows a distribution significantly different from chance (p < .01).No engineer has poorest knowledge in the intuitive mode. Thus we concludethat if the attenuation of ra by the inconsistency in the task and in eachsubject's performance were eliminated, most engineers would have achievedhigher accuracy of prediction in the intuitive mode than any othe'. Mostengineers would have judged least accurately in the analytical mode. Thenumber of engineers whose quasi-rational judgments were best would have beenabout the same as the number whose quasi-rational judgments were poorest.

Cognitive control. The cognitive control component, Rs, is of coursesignificantly higher (p < .001) in the analytical mode than in the other modesbecause the answers in the analytical mode were mechanically generated fromthe engineers' formulas. Far more engineers had a larger Rs in thequasi-rational mode than in the intuitive mode (16 of 21; p < .05). Therelation between cognitive control and repeated trials reliability isdiscussed in Appendix E.



33/83

Hammond, Hamm, Grassia, and PearsonTable 6

Number of Engineers with Highest Knowledge (G) in Each Condition

Is t Urder

13 I >0>A 9I >A> 4

6 >>I>A 6[Q>A>I 0

A 2 A>I>Q 2A > >1A > Q > I 0

21 21X= 8.86, p < .02 X= 18.14, p < .01

Summary of findings regarding achievement. Fewest engineers wvere mostaccurate (ra) in the intuitive mode, although this finding is notstatistically significant. Comparison of knowledge as measured by G, however,indicates that even though the engineers explicated their knowledge andapplied it rigorously, i.e., with perfect consistency, in the analytical mode,their achievement would have been higher in the intuitive an d quasi-rational

SW-7 -7

191Direct Comparison Page 30


34/83


modes if they had been equally consistent there. This result supports theconclusion that engineers possess and can apply implicit knowledge abouthighway safety that they do not accurately express in rigorous, retraceable,quantitative form.

Differential Distributions of Errors under Different Cognitive Modes

Investigating the covariation between judgment and criterion does notexhaust the possible methods for evaluation of performance. Brunswiksuggested (1948, 1956, pp. 89-91; see also Hammond, Note 1) that the use ofdifferent modes of cognition may lead to similar achievement but to verydifferent types of errors. Based on the distinction between intuition andanalysis described by Brunswik and Hammond, we test the following hypotheses:

401. Errors made in the intuitive mode are normally distributed and also

less frequently exactly correct and less frequently widely incorrectthan errors made in the analytical mode;

2. The quasi-rational mode produces an error distribution that liesbetween the distribution of errors in intuitive and analyticalcognition.

Rescalinq procedure. The error scores were produced by subtracting theaccident rate for a highway from the engineer's judgment. The response scaleused in the intuitive mode ran from 1 ( unsafe) to 10 (= safe), whereas thescales in the quasi-rational and analytical mode ran from 0 to 32 accidentsper million vehicle miles traveled. To make the error distributions

-1

- - * -



35/83

*. Hammond, Hamm, Grassia, and Pearson

comparable, judgments made in the intuitive mode were rescaled onto the 0 to32 scale. Further, some engineers' analytical formulas did not producenumbers between U and 32. Their responses were rescaled by mapping the numberthey intended their formula to produce for the safest road onto zero, and thenumber they intended fo r the most dangerous road onto 32 .

Hypothesis 1: Th e error distributions in the analytical condition shoulddeviate from normal more often than they do in the intuitive condition.

Procedure. For each engineer and fo r each mode, the mean and standarddeviation of each engineer's error distribution were used to produce a normaldistribution. Th e null hypothesis is that the observed distribution of anengineer's errors does not differ significantly from the constructed normaldistribution. Therefore, the normal distribution was divided into sixcategories expected to have equal numbers of observations (the boundaries werelocated at -. 97, -. 43, 0, +.43 and +.97 standard deviations away from themean), and the number of the errors that fell into each category was observed.

Results. In the intuitive, quasi-rational and analytical modes therewere few engineers (two, five, and three out of twenty-one, respectively)whose distributions of deviations differed significancly f.'om normal.Additionally, a comparison of the chi-squares for the deviations of theseerror distributions from normal revealed that no mode tended to have larger

0 deviations than the other modes. These results offer no support for thehypothesis that the analytical mode would produce more non-normaldistributions than the intuitivE mode.



36/83

Hanunond, Hamm, Grassia, and Pearson

Hypothesis I also asserts that more of the engineers' analyticaljudgments than intuitive judgments should have error distributions withpositive kurtosis; the distributions of errors from quasi-rational judgmentsshould lie between those of the other modes.

Procedure. Since a positive kurtosis indicates a peaked frequencydistribution, i.e., one in which the answers are often nearly correct yetoccasionally very far off, we examined the degree of kurtosis in eachengineer's error distribution.

Results. Th e results in Table 7 indicate that the error distribution forthe analytical mode was nmore peaked than for the intuitive mnode for seventeenof the 21 engineers (p < .01). Th e quasi-rational error distribution was morepeaked than the intuitive error distribution for seventeen engineers(p < .01). However, for only ten of the engineers was the analytical errordistribution more peaked than the quasi-rational. Th e results fromexamination of kurtosis thus support the hypothesis that more engineers would W1produce answers that were less frequently exactly correct in the intuitivemode than in the analytical mode. Finding that the quasi-rational errordistribution was not different from the analytical was unexpected.

A second way of testing the hypothesis concerning the shape of errordistributions in each cognitive mode is to look at each engineer's performanceas a whole rather than at each answer individually. If the formula iscorrect, achievement (ra, the correlation between answers and accident rate)should be relatively high because the formula should generate more answersthat are nearly correct than intuitive judgments do. If an engineer makes anerror in a formula, all answers should be wrong and ra should be relativelylow.



37/83

Hammond, Hamm, Grassia, and PearsonTable 7

Number of Engineers with Various Pairwise Orders ofKurtosis of Error Distribution

A > I 17 X2 6.86I > A 4 p < .01

*2A > 10 X2 O0OQ> A 11 NS

> I 17 x 6.86I >Q 4 p < .01

In Table 8 the range of correlations (r) between answers and accidentrates for the five best performances in the analytical mode (.643 to .731) andfor the five best performances in the intuitive mode (.618 to .636) show no

* overlap. From this standpoint the best analytical performances are clearlysuperior to the best intuitive performances. However, the median achievementin the analytical mode was .467, far worse than the median achievement in theintuitive mode (.576); and the analytical achievements covered a total rangeof .957 a3 compared to the intuitive range of .565. Engineers' performancedata as well as analysis of error on individual judgments thus supports the

O hypothesis that analytical cognition is more often very precise yet more oftenwidely in error.

ti



38/83

Table 8Achievement in Each Mode

Intuitive Quasi-rational AnalyticalSpan of Best FiveEngineers' Achievements .618 - .636 .624 - .738 .643 - .731Median Achievement(all 21 engineers) .576 .516 .467Range of Achievement(all 21 engineers) .565 .738 .957

Experimenter's Observations Regarding "Widely Incorrect" Answers.The above analyses describe the distributions of errors produced by

different modes of cognition; but the "catastrophic" errors that Brunswikpredicted would occur in the analytical mode are best illustrated by examplesof errors made by three engineers in the analytical mode: 5

1. Engineer #2 made a careless arithmetic error in producing weights inhis formula. He first assigned a weight of .10 to each of the tei, cues. Nexthe adjusted the weights of important cues to .12. Finally, intending toassign weights of .08 to cues he felt were slightly less important, he wroteinstead .8. Thus he gave the highest weight to the cues to which he wished togive least weight. The effects of his error were "catastrophic": hisachievement (ra) was .071, and his mean error was 44.176 on a scale heaintended to go from 0 to 32.

Direct Comparison Page 35 1Hammond, Hamm, Grassia, and Pearson


39/83

2. Although provided with the information that certain cues could assumea value of zero, Engineer #4 produced a formula in which some of these cuesappeared in the denominators of fractions. As a result, his answer wasindeterminate on 5 of the 39 highways.

3. Engineer #10 produced a linear weighted average formula, in which heinadvertently used the wrong sign on two cues. His achievement wasconsequently a very low -. 048. Had he not made this error his achievementwould have been .293.

Since the processes of intuitive and analytical cognition cannot bedirectly observed, we cannot explain occasional poor performance in thefilm-strip and bar-graph tasks by citing the kind of careless, one-time errorsin execution of a complex symbolic procedure as we did above. It is unlikelythat this kind of error is occurring in the intuitive and quasi-rationalmodes, however. Even if, in the film-strip and bar-graph tasks, the engineerswere actually engaging in a step-by-step cognitive procedure similar to thatobserved in the analytical mode, they would have to make the same error duringeach trial in the task. Although a careless error is guaranteed to occur withcomplete consistency when a faulty formula is applied to thirty-nine highways,it is not guaranteed by the conditions of intuitive and quasi-rationalcognition. On the contrary, since no engineer made perfectly consistentjudgments in either the intuitive or the quasi-rational modes, none could beexpected to make consistent errors. Poor performance in the intuitive andquasi-rational modes must instead be attributed either to an engineer's lackof knowledge or to his inability to bring his knowledge accurately to bear onthe task.

Ul

Direct Comparison Page 36 0Hammond, Hamm, Grassia, and Pearson


40/83

Summary regarding differential errors. When the distribution of errorswas examined, the results provided no support for the general hypothesis thaterrors would be found to be normally distributed in the intuitive mode moreoften than in the analytical mode, with errors in the quasi-rational modeintermediate. However, the predicted differences in kurtosis were observed:engineer's performances in the analytical mode were more often more accuratethan their performances in the intuitive mode, but occasional large errorswere produced in the a,,alytical mode. Results in the quasi-rational mode werelittle different from the analytical mode in this regard. Inspection of theengineers' analytical work revealed instances of large errors as anticipated(misplaced decimals, division by zero, and reversed signs). No similar errorswere found in the other modes of cognition.

On e reason that the anticipated differences in the degree of normality of 'Ierror distributions were not found may be that the formulas produced by theengineers were quasi-rational in form; that is, many engineers developedformulas that were analogous to a robust weighted average (eleven by a strictcriterion, twenty by a loose criterion; see Appendix A). Formulas of thistype (when applied without errors in calculations) are as unlikely to producelarge errors as the weighted averages that the evidence suggests '.he engi.neerswere using in the intuitive and quasi-rational modes.

IuD

oI

Direct Comparisor Page 37Hammond, Hamm, Grassia, and Pearsonz


41/83

Confidence

Ansvier Confidence Compared with Method ConfidenceCognitive continuum theory predicts that subjects will be more confident*

in answers produced by intuitive cognition than in answers produced byanalytical cognition. Conversely, the theory predicts mnore confidence in themethod of analysis than in the method of intuition. These predictions may betested by analyzing (a) the order of engineers' confidence among modes ofcagnition, for answer and method confidence separately, and (b) the order ofengineers' answer confidence and method confidence within each mode. Thesetwo approaches are diagrammed in Figures 3 and 4.

* Intuitive Quasi-rational AnalyticAnswer Confidence-m m-- -: '

Method Confidence

Note: The direction of the arrows indicates decreasing confidence.

Figure 3. Predicted Relative Confidence for Each Pair of Cognitive Modes



42/83

Figure 3 illustrates the expected order of answer confidence and ofmethod confidence among modes of cognition. Answer confidence should berelatively high in the intuitive mode and decrease as cognition becomes moreanalytical. The reverse should be true for method confidence.

Figure 4 represents the expected within-mode order of engineers' answer* confidence and method confidence. In each pair of modes engineers should

express more method confidence than answer confidence for the mode closer tothe analytical pole on the cognitive continuum.

N

Q IQ-Answer Confidence

Method Confidence 1 iNote: The direction of the arrows indicates decreasing confidence.

Figure 4. Predicted Relationship Between Answer Confidence and MethodConfidence for Each Pair of Cognitive Modes

Procedure.

*Q All confidence ratings in this study were indicated on a 1-to-IO scalewhere 10 = high confidence. In the intuitive and quasi-rational modes, answerconfidence is indicated for each engineer by the mean of the confidence

*ratings he made for each of his judgments. Answer confidence in the

0~



43/83

analytical mode and method, confidence for all th~ree modes are represented by the means of responses to particular confidence questions (see Appendix F)which the engineer answered after making judgments or constructing a formula.

Results

Orders of Answer an d Method Confidence. The rank order of engineers'confidence across modes is shown in Table 9. For answer confidence the modalorder (10 engineers) was exactly the predicted order (see Figure 3): more -1subjects indicate more answer confidence in the intuitive mode than in thequasi-rational mode, and more in the quasi-rational than in the analyticalmode. As predicted, twenty engineers had more answer confidence in theintuitive than in the quasi-rational mode (p < .001); and sixteen had more inthe intuitive than in the analytical mode (p < .05); however, only eleven hadOWmore answer confidence in the quasi-rational than in the analytical mode (NS).

For method confidence, the modal confidence order was I > A > Q, whichwas not predicted. Thirteen an d a half engineers (ties counted as 1/2 in eachof the indicated orders) had more method confidence in the analytical modethan in the quasi-rational mode (NS, p < .20); but only three had more methodconfidence in the analytical than the intuitive mode (p < .01 i, the wrongdirection), and only 3.5 had more method confidence in the quasi-rational thanin the intuitive mode (p < .01 in the wrong direction).

* L

Direct Comparison Page 40Hammond, Haemm, Grassia, and Pearson


44/83

Table 914Count of Engineers Who Had Each Possible Confidence

Order among the Three Modes

Answer Confidence Method ConfidenceI > > A 10 3I > A >Q 5 11.5*Q> I > A 1 2.5*- A>I 0 1A > I > 5 2

- A>Q>1 0 0

2= 22.14, p < .001 = 25.78, p < .001* When an engineer had equal confidence in tw o mndes, .5 countwas assigned to each cf the two orders. For method confidence,there were two ennineers who had ties. Method Confidence datawas missing for one engineer.

*:. Meth:.d Confidence Compared with Answer ConfidenceComparison of each engipeer's method confidence with his answer

confidence shows that the majority of engineers had greater confidence inanswer than in met.hod for the intuitive and quasi-rational modes and greaterconfidence in method than in answe'r for the analytical mode (see Table 10).Chi-squared tests showed, however, that this finding is not significant,

Direct Comparison Page 41Hammond, Haemm, Grassia, and Pearson


45/83

Table 10Relative Answer and Method Confidence for Each Cognitive Mode*

I i A''LAnswer Confidence is greater 12 15 7

Equal 2Method Confidence is greater 8 6 12

*Due to missing data, N = 20 for the intuitive task.

In sumhary, the prediction that the engineers would have more confidencein their answers in the intuitive mode was born out, but the prediction that Ithe eng;neers would have more confidence in their method in the analyticalmode was contradicted. In the comparison of answir and method confidencewithin mode, the predicted pattern of greater method confidence than answerconfidence in the analytical mode and greater answer confidence in theintuitive ,rode was observed, but did not teach statistical significance. Mostengineers had greater confidence in the intuitive mode for both answers andmethod. Judging safety was less susceptible to analytical cognition then wehad anticipated.



46/83

Appropriate Placement of Confidence

Appropriate placement of confidence can be evaluated in terms of whetherexperts' confidence in their judgments reflects their achievement--that is,are they more confident in the mode in which their achievement is higher?

Table 11 displays for both answer confidence and method confidence therelation between relative confidence and relative achievement for each pair ofmodes (intuitive versus quasi-rational, intuitive versus analytical, andquasi-rational versus analytical).

In each of the four-celled blocks, the upper left and lower right cellscontain the number of engineers whose confidence was appropriate given their"achievement; the upper right and lower, left cells, engineers whose confidencewas misplaced. For example, consider the block relating relative answerconfidence to relative achievement for the intuitive and quasi-rational modes.Of the thirteen engineers who had higher intuitive achievement than"quasi-rational achievement, twelve of them had appropriate answer confidence;that is, they had higher answer confidence in the intuitive mode than in thequasi-rational mode. However, none of the eight engineers who achieved higherin the quasi-rational mode than the intuitive mode had appropriata confidence.

.O'

Direct Comparison Page 43Hammond, Harmm, Grassia, an d Pearson


47/83

-C Va.) A 00. LLJ A ODLn U.< M V rU '-( .WA L~AJ 0 00o 0 ',-~4- S. a -); &u-a)4SA A A L 0 )-

0' 1 0. 0

cl C'l &Z- (M:

( V

v f,,. Ia,.'-

M4-)'0j 0*A A A A 0a) i.

. < .' -a!

- *- 4A .0 0L

. a., >0 0-40-4 C 0 M4 -1 0

4C) * 4 . *)C., - V LI.. .044C LAJ u. U M. 4..4-O C4. .UI C C 4-) (W 11It V E W 14 1S r- "-I La.. U-. r.a.. 4j - (4-. 002 S 4-;,) C- D 4-" 0 ) ,4,-b 0.U 0 L..)() C (j r-. a4-A a) 0A 100. .. ... ".....) ,. - . - 0 . . - -EUJ ^4a C0,- 4-)-' 4- 39 X___ _ _ 0 X.___ __ )O CC 4A. EGI0 C 4.)a) Uz04.)C M M* 4- cc0 u- "a0 nU $-. (a 4) r_A A .0A A 0 0) = (a4-Cr 4-'En-c -ccE.4*aL M, C (DC (1) 4-,-

L E'~-EU 4IV 0 u 4 Q

0.0 UCa


48/83

Both answer and method confidence revealed a pattern of overconfidence inthe intuitive mode as compared to the quasi-rational mode (p < .001 andp < .01, respectively). Most engineers also had inappropriately high answerconfidence (p < .10) and method confidence (p < .01) in the intuitive modewhen compared to the analytical mode. No pattern of overconfidence was seenbetween the quasi-rational and analytical modes, although the engineers werevery inaccurate. In fact, a null model of random confidence judgments wasrejected here because the order of engineers' performance is the opposite ofthe order of their confidence when the quasi-rational and analytical modes arecompa2ed.

Since the engineers received no feedback about their achievement (ra),

the appropriateness of their answer confidence and method confidence must alsobe evaluated in terms of knowledge (G). The same pattern of overconfidence inthe intuitive mode that was observed with achievement is seen with knowledge(Table 12).

Summary Regarding Aprpriateness of Confidence

When appropriateness of confidence is evaluated in terms of empiricalachievement (r ) most engineers were found to be (a) more confident aboutatheir answers and also about the method used to produce their answers in theintuitive mode than in the other two modes; and (b) overconfident inintuitive cognition and underconfident in analytical and quasi-rationalcognition. The general pattern of overconfidence in intuitive cognition andunderconfidence in analytical cognition remains when the engineers' knowledge(G) is examined.

Direct Comparison Page 45Hammnond, Harmm, Grassia, and Pearson


49/83

LaJ A M- LUJ A C)EV) (.) 4-LUJ LUJ 0 00

.'-4 A 6-4 - C ILa.. U-. - 0o *'c S-0 -LC;.. C..) %- ()) (v b."A Lfl ' A MV 0 Q) '- 6

0'y0 0-i___ X_ r- r-srQ)-

A A A A0) -CD-'

lid 0K....Jwa0 ..W e >a I J LZ0 L"U Q$LC 4).)0 0 (U

C~000. IA

LU~~ ~,0A (" ) 0) S-.'-~~~~4 U C 1 ' )0 )Jcm LAJ A. U LUJ A. M CD ca.I - 0 u 0C 0 4 J

4J LL. CL) Q L&a.. =.04-~ CUC'.J 0) ) 3 M 4J. 4)-4 u C CD WC * 4- 0M )4J -0a 0 u) 4 C C-) 44-u)0)L A C14 0 A LOl Cjr-V v-L 4I -U. CA ).,- $- -4 - 00 -EU4- 0) 04J V0 IC% Cr J0 01 rC-I

r- 4J' 0)4- 0 4-). OM0 0C'C- O

EnA A .0 d. :5.EC 0~ C (V

IA C -4J0(Af 4) 0) COC. MU4-CL (a-. 0) C

04 EU r- U 0)+ U0 10 tv Cc-U

4'I IL *u-) 0 *rL-S-LC.) AU- V La) A) 0) +J L L

.0 14 ' 4- V)OK)C=0 4U- U (A -~r-. EU..C.) _-y I 0~ S0r0)tC

A '0A '- 4 LOC u)( S.- - 0) C0 0:CN. 0 S.-S- 3: (

4-- ) Ln toa C V -)0. -t-

A A A A 0 CL M CLM0()4='



50/83

Sunwnary and DiscussionIn an effort to broaden and deepen comparisons of the efficacy of

intuitive and analytical cognition six criteria were specified: (a) direct"comparisons should be made between intuitive and analytical cognition; (b)the ;ubjects used in the study should be analytically competent by virtue of

:- their professional training; (c) the concepts of intuition and analysis, aswell as the task properties that induce each, should be given clear meaning-(d) substantive and methodological cognition should be included in the samestudy; (e) conclusions regarding the relative efficacy of these modes ofcognition should be norm-independent, and thus not subject to disputedstandards of inductive inference; and (f) conclusions should not berestricted to problems involving the logic of the standard probabilitycalculus.

These criteria were met by (a) requiring each subject to engage inintuitive and analytical cognition with regard to the same problem, (b)employing professional highway engineers as subjects, (c) specifying theprincipal characteristics of intuitive and analytical cognition and theproperties of the tasks that induce them, (d) distinguishing betweenmethodological and substantive cognition, (e) valuating performance in termsof empirical achievement, and (f) employing problems that do not require theuse of the conventional probability calculus.

Direct Comparison Page 47Hannond, Ham, Grassia, and Pearson


51/83

The results indicate that within these broadened conditions:1. Achievement in a cognitive mode is greatly influenced by the

degree of consistency present. The absolute consistency of analyticalcognition affords it an advantage in empirical achievement over intuitiveand quasi-rational cognition, even when the substantive knowledge that isemployed analytically is relatively less correct. Conversely, theinconsistency (low cognitive control) of intuitive cognition will lead toan underestimate of its empirical value, unless such inconsistency isidentified and removed.

2. The knowledge implicit in intuitive judgments was found to beempirically superior to quantified, formalized knowledge. Wheninconsistency was removed from the engineers' intuitive an d quasi-rationaljudgnents, intuition an d quasi rationality were more efficacious thananalysis for most of the subjects.

3. Analytical cognition was more often highly accurate, yet moreoften very inaccurate when compared with intuitive cognition for most ofthe subjects. Large errors were often made in the analytical mode, butseldom in the intuitive or quasi-rational modes.

4. When performance is evaluated in terms of achievement, engineerswere most confident in the intuitive mode, not only in their answers (asanticipated) but in their method as well (not anticipated). However, thisgreater confidence in the intuitive mode was not always warranted.

'a



52/83

These results demonstrate that a cognitive continuum theory has positiveutility: it enables the findings of this study concerning achievement,

- knowledge, errors, and confidence to be largely predicted from the nature ofthe task properties employed. This conclusion has implications for cognitiveresearch as a whole. Research ranging from the intuition-inducing tasks ofjudgment and decision making to the analysis-inducing tasks of problem solvingmay also be understood to be located along a cognitive continuum.Specification of task properties and description of cognitive modes may wellexplain a variety of results in cognitive research and thus reduce the currentisolation of research approaches. It may well turn out that, like theengineers in our study, we know more about our field than we have previouslybeen able to express.

:I

m Direct Comparison Page 49Hammond, Hamm, Grassia, an d PearsonReference Notes


53/83

1. Hammond, K. R. Unification of theory an d research in judgnent anddecision making (Center for Research on Judgment and Policy).Unpublished manuscript, University of Coloradc, 1982.

II

9

Direct Comparison Page 50Hammond, Hamm, Grassie, and Pearson


54/83

References

Adelman, L. The influence of formal, substantive, and contextual taskproperties on the relative effectiveness of different forms of feedbackin multiple-cue probability learning tasks. Organizational Behavior andHuman Performance, 1981, 27, 423-442.

Anderson, B., Deane, D., Hammond, K. R., McClelland, G. H., & Shanteau, J.Concepts in udgment and decision research: Definitions, sources,interrelations, comments. New York: Praeger, 1981.

Brehmer, B. Policy conflict as a function of policy similarity and policycomplexity. Scandinavian Journal of Psychology, 1972, 13 , 208-221.

Brehmer, B. Social judgment theory and the analysis of interpersonalconflict. Psychological Bulletin, 1976, 83, 985-1003.

Brehmer, B. Preliminaries to a psychology of inference. ScandinavianJournal of Psychology., 1979, 21. 209-221.

Brehmer, B., & Hammond, K. R. Cognitive factors in interpersonal conflict.In D. Druckman (Ed.), Negotiations: Social-psychological perspectives.Beverly Hills: Sage, 1977.

"Brooks, L. Nonanalytic concept formation and memory for instances. InE. Rosch and B. Lloyd (Eds.), Cognition and categorization. New York:Erlbaum, 1978.

Direct Comparison Page 51 *Hammond, Harm, Grassia, &nd PearsonBruns'ik, E. Th e conceptual framework of psychology. In International


55/83

encyclcpedia of unified science (Vol. 1, No. 10). Chicago: Universityof Chicago Press, 1952.

Brunswik, E. Perception and the representative des-gn of psychologicalexperiments (2nd ed.). Berkeley: University of California Press, 1956.

Cohen, L. J. Can human irrationality be experimentally demonstrated? TheBDhovioral and Brain Sciences, 1981, 4(3), 317-370.

Duda, R., & Shortliffe, E. Expert systems research. Science, 1983, 220,261-268.

Edwards, W. Bayesian and regression models of human informationprocessing--a myopic pe'spective. Organizational Behavior and Human

0 Performance, 1971, 6, 639-648. -.

Einhorn, H. J., & Hogarth, R. M. Behavioral decision theory: Processes of-P judgment and choice. Annual Review of Psychology, 1981, 32 , 53-88.

Einhorn, H. J., & Hogarth, R. M. Reply to commentaries. In G. Ungson & D.Braunstein (Eds.), Decision ,making: An interdisciplinary inquiry.Boston: Wadsworth, 1982.

Fisch, H. U., Hammond, K. R., &Joyce, C. R. B. On evaluating the severityof depression: An experimental study of psychiatrists. British Journalof Psychiatry, 1982, 140, 378-383.

mmm

Direct Comparison Page 52Hammond, Hanun, Grassia, and PearsonFox, J. Making decisions uwder the infltuence of memory. Lsychological


56/83

Review, 1980, 81, 190-211.

Gillis, J. S. Interpersonal conflict: Effects of chlorpromazine andthiothixene on acute schizophrenic patients. In K. R. Hammond & C. R. B.Joyce (Eds.), Psychoactive drugs and social judgment: Theory andresearch. New York: Wiley, 1975.

Gillis, J. S. Sources of judgmental impairment in paranoid and non-paranoidschizoohrenics. Journal oF Abnormal Psychology, 1978, 87, 587-596.

Hammond, K. R. Representative versus systematic design in clinicai. psychology. Psychological Bulletin, 1954, 51, 150-159.

Hammond, K. R. Probabilistic functioning and the clinical met~iod.Psychological Review, 1955, 62(4), 255-262.

Hamn,ond, K. R. (Ed.). The psychology of Egon Brunswik. New York: Holt,Rinehart, & Winston, 1966.

Hammond, K. R., & Brehmer, B. Quasi-t'ationality and distrust: Implicationsfor international conflict. In L. Rappoport & D. Sunmmers (Eds.), Humanjudgment and social interaction. New York: Holt, Rinehart, and Winston, qP11973.

Hammond, K. R., McClelland, G. H., & Mumpower, J. Human judgment anddecision makn Theories, methods, and procedures. New York: Praeger,1980.

Ji~m l!!-~lllllilliillliH~ll/4

Direct Comparison Page 53 .0Hammond, Hamm, Grassia, and PearsonHammond, K. R., Stewart, T. R., Brehmer, B., & Steinmann, D. 0. Social


57/83

judgment theory. In M. F. Kaplan & S. Schwartz (Eds.), Human judgntmeritand decision processes. New York: Academic Press, 1975.

Hammond, K. R., & Summers, D. A. Cognitive control. Psychological Review,1972, 79, 58-67.

Hammond, K. R., & Wascoe, N. E. (Eds.). New directions for methodology- ofsocial and behavioral science: Realizations of Brunswik's representativedesign. San Francisco: Jossey-Bass, 1980.

Hastie, R. Social inference. Annual Review of Psychology, 1983, 34,511-542.

Holzworth, J. Intervention in a cognltive conflict. Organizational Behaviorand Human Performance, in press.

Johnson-Laird, P., & Wison, P. A theoretical analysis of insight into areasoning task. In P. Johnson & P. Wason (Eds.), Thinking: Readings incognitive science. Cambridge, England: Cambridge Univers-ty Press,1977.

Jones, C. &Harris, P. insight into the law of large numbers: A comparisonof Piagetian and judgment theory. Quarterly Journal of ExperimentalPsychology, 1982, 34a, 4/9-488.

Kahneman, D., Slovic, P., & Tversky, A. Judgments under uncertainty:Heuristics and biases. New York: Cambridge University Press, 1982.

'4

..... .... ........ .... .... ....

7, - ' - - -- 7'-. ---

Direct Comparison Page 54Hammond, Hamm, Grassia, and FearscnKahneman, D., & Tversky, A. Oi the study of statistical intuitions.


58/83

Coqnit.ion, 1982, 11, 123-'41.Kirwan, J., Chaput de Saintoige, U., Joyce, C. R. B., & Lurrey, H. Clinical

judgment dnalysis: Practical application in rheumatoid arthritis.British Journal of Rhematology, in press.

Kyburg, H. The logical foundations of statistical inference. Dordrecht,Holland: D. Reidel, 1974.

Kyburg, H., & Smokler, H. Studies in subjective probabi!lit (2nd ed.). NewYork: Wiley, 1980.

Larkin, J., McDermott, J., Sinon, D. P.. & Simon, H. A. Expert an d noviceperformance in solving physics problems. Science, 1980, 208(4450),1335-1342.

Levi, I. The enterprise of knowledge. Cambridge, MA. MassachusettsInstitute of Technology Press, 1980.

Lichtenstein, S. , Fischhoff, P., & Phillips, L. Calibration ofprobabilities: The state of the art to 1980. In ). Kahneman, P. Slovic& A. Iversky (Eds.), judgment under uncerta;nt,: Heuristics and biases.Cambridge, England: Crarbridge Univers'ty Press, 1982.

Luce, R. D. Individual choice behavior. New York: Wiley, 1959.Oskamp, S. Overconfidenct; in case-study judgments. Journal of Corsulting

Psycholo_, 1965, 29, 261-265.

Direct Comparison Page 55Hantnond, Haemm, Grassia, and PearsonScribner, S. Modes of thinking and ways of speaking: Culture and logic


59/83

i* reconsidered. In P. Johnson-Laird & P. Wason (Eds.), Thinking: Readingsin Sp4nitive science. Cambridge, MA: Cambridge University Press, 1977.

Shanteau, J. An information-integration analysis of risky decision making.In M. F. Kaplan & S. Schwartz (Eds.), Humen judgment and decisionprocesses. New York: Academic Press, 1975.

0 Slovic, P., Fischhoff, B., &Lichtenstein, S. Behavioral decision theory.Annual Review of Psychology, 1977, 28, 1-39.

"Slovic, P., & Lichtenstein, S. Comparison of Bayesian and regressionapproaches to the study of information processing judgment.Or-ganizational Behavior and Human Performance, 1971, 6, 649-744.

Tversky, A. Elimination by aspects: A theory of choice. PsychologicalReview, 1972, 79, 281-299.

Twerney, R., Doherty, M., & Mynert, C. (Eds.). On scientific thinking. NewYork: Columbia University Press, 1981.

Wason, P., & Johnson-Laird, P. Psychology of reasoning: Structure andcontent. Cambridge, MA: Harvard University Press, 1972.

_.

_-, _____ ._.______"_,_"_ "' " "__-" "_ "L " -'' ". "x -'. ,". ." ' V' "' . . .

* Direct Comparison - Appendix A Page 1Hammond, Hamnm, Grassia, and PearsonAPPENDIX A


60/83

Characteristics of Engineers' FQrmulas forSafety, Analytical Mode

Eleven engineers developed formulas that involved only additivecombinations of the cues (median ra = .467). Four of these involved onlylinear functions of the cues (median ra 281). For example: iAccident Rate = 12.8(23 - (SWIDTH + LWIDTH))/15 +12.8(OBSPM + CURVEPM + INTPM)/29 +6.4(TRAFMIX + PCTNPZ)/115One expressed nonlinear functions with graphs (ra = .518); six used tables toexpress nonlinear functions of cues (median ra = .496).

Another engineer used only a multiplicative combination of variables(ra .670):

Accident Rate .25(Fs* FL * F~p FG R FIP FLW FNpZ FGR FIPMFOPM * FASL * TV)

* where each F is a one-dimensional table expressing a linear or nonlinearfunction of the cue.

The remaining nine engineers produced formulas that involved a hierarchy*of organizing principles (median ra = .436). For example, one configurally

combined a subset of factors in a table and then added the result togetherwith other factors. Five of these hierarchical formulas used adding and

multiplying (median ra = .436). For example:Accident Rate = CURVEPM + (60 - AVESL)/1O -

(LWIDTH + SWIDTH - 23)/2 +(TRAFVOL/10000)(INTPM + OBSPM - 3) +(TRAFMIX * GRADE * PCTNPZ)/10000Three engineers used adding as well as tables (median ra .390). One

aaused multiplying and tables (r-a .641).

-0

Direct Comparison - Appendix B Page 1Hammond, Ham, Grassia, and PearsonAPPENDIX B


61/83

Maximal Guidance Procedure

This procedure was designed to give the engineer a maximum of guidance inconstructing his formula for safety in order to prevent the commission ofminor errors and to ensure that his formula adhered to principles ofmeasurement theory with which he might not be familiar. The procedure 0consisted of the following steps:

-- Specify the answer scale. -- Specify the scale for each input dimension and its overall relation to

the answer scale, and identify possible interactions with otherdimensions.

Group the input dimensions according to their redundancy, similarity,or mutual interactions.

-- Express the formula as a hierarchy of groups of variables.-- Determine what organizing principle should be used at eacn level of

hierarchy.-- Specify the function form governing each dimension's input to its

organizing principle.-- Combine all the above information into one formula.

98

Direct Comparison Appendix B Page 2Hammond, Hamm, Grassia, and PearsonThe engineer was guided through these steps by a series of forms which


62/83

contained instructions for the steps an d choice points, and on wnichintermediate steps were recorded. Tw o examples follow. The engineer alsoreceived detailed tutorials about interactions an d organizing principles aspart of the maximal guidance procedure.

kit.,

Direct Comparison - Appendix B Page 3Hammond, Hamu, Grassia, and PearsonForm 1: Answer Dimension Form


63/83

Name Task _.Answer dimension's units .Range of possible answers: Low High

A "natural V" on a scale means that when it is called "0" there really isNONE of the quality being measured. If you had a natural 0, hen it wouldmake sense to say that an "8" is twice as much of a thing as a "4"; but ifthe 0 was arbitrary, it wouldn't have that sort of meaning.For example, if yo u have savings of $10,000, you have twice as much moneyas if yo u had $5000, because 10000 is twice 5000. Here the $0 is a natural 0.But 32 degrees F is not twice as warm as 16 degrees F, because the 0 on thetemperature scale is picked arbitrarily. In other words, it does not have a

natural 0.Does the answer dimension have a natural 0? Yes No .____It is useful, when considering numbers that measure a dimension, to askwhether the intervals between the numbers have consistent meaning, or whetherthe numbers simply express order. For example, is the difference between a

Direct Comparison of Intuitive, Quasi-Rational and Analytical Cognition; Hammond.pdf

Documents

Transcript of Direct Comparison of Intuitive, Quasi-Rational and Analytical Cognition; Hammond.pdf