HUM AN FA C TOR S 198224(1)61-73

Pilot Judgment Training and Evaluation

RICHARD S JENSEN Department of Aviation The Ohio State University Columbus Ohio

An analysis of accident statistics reveals that over 50 of pilot-caused civil aviation acci-dent fatalities are the result of faulty pilot judgment Although the FAA requires examiners toevaluate pilot judgment it provides no definition or criteria against which such an evalua-tion can be made In spite of the statistics implicating pilot judgment in many aviationfatalities attempts to teach it are almost nonexistent It is but a slight overstatement to saythat good pilot judgment is learned by the lucky and the cautious over many years of variedflying experiences This paper examines some of the decision-research literature in an at-tempt to provide an operational definition of pilot judgment and to suggest ways that pilotjudgment may be taught and evaluated in civil aviation

INTRODUCTION

From the beginning of aviation historypilots have been expected to exercise a con-siderable amount of judgment in the overalltask of flying an airplane However in recentyears increasing societal demands for safetydependability economy effectiveness andreduced energy consumption have increasedthe complexity of civil and military flying op-erations thus magnifying the pressures forgood pilot judgment Furthermore tech-nological advances that have eased muchof the pilots burden for precise aircraft con-trol have not greatly eased the pilotsdecision-making workload In many casesthese advances have only created demandsfor higher levels of skill knowledge andjudgment for which few pilots have beentrained and the costs to train them to operateeffectively in this changing system are be-coming prohibitive

If it were merely a matter of teaching flying

I Requests for reprints should be sent to Dr Richard SJensen Department of Aviation The Ohio State Univer-sity Box 3022 Columbus OH 43210

skills the task of training pilots to operatesafely in our complex aviation system wouldbe a much smaller one than it is Unfortu-nately because actual conditions are neverquite the same as those used to develop avia-tion regulations procedures and perfor-mance limitations the safety of a given flightalso depends upon a significant amount ofevaluation and interpretation of existingconditions by the pilot

For example the conditions used to de-velop flight performance values for a partic-ular type of airplane may be ideal eg cleanairplane surfaces a new engine a new prop-eller an unrestricted air filter and a com-pany test pilot In actual conditions the pilotmust compare these book values obtained inideal conditions with those actually being en-countered These actual conditions may in-clude a dirty airplane a slightly used enginea few marks on the propeller a somewhatdirty air filter and a less than perfect pilotThe pilot must then evaluate many otherconditions such as gross weight center ofgravity wind temperature humidity al-titude etc for comparison with those used in

copy 1982 The Human Factors Society Inc All rights reserved

62--February 1982

the book to determine the expected flightperformance Finally the present and fore-cast weather the terrain and the expectedtraffic density must be checked These arethen compared with the pilots estimate ofher or his own capability before determiningwhether or not the planned flight will be safe

Examples of decisions being made withimperfect information can be found in allareas of flight activity Furthermore everydecision that the pilot makes is colored byphysiological psychological and social pres-sures that are virtually impossible to weighproperly on the spot For example just aspersons watching a sporting event may seean infraction or foul differently dependingupon their vantage points and which teamthey support a pilot may be influenced toview the weather outlook or his or her ownabilities differently depending upon the im-portance or value he or she assigns to a givenflight Evidence from the social psychologyliterature (Kogan and Wallach 1964)suggests that ones self-image and the need tomaintain an external image largely deter-mine how much effect the costs and payoffsfor a given flight will have on the judgment ofones own ability to make a safe flight

Some pilots may be susceptible to socialpressures that result in less than rationalpilot judgment Irrational pilot judgment hasbeen exhibited by such unsafe practices asflying under bridges landing on busy high-ways attempting to land in football stadi-ums and flying formation on other un-suspecting pilots Potential sources of so-cial pressure that may lead to these types ofactivities include peer reactions fear of fail-ure censure from superiors or family mem-bers and many other factors (Janis andMann 1977)

The Problem

An analysis of accident statistics bycategories of pilot activities reveals the seri-

HUMAN FACTORS

ous nature of the problem Most analyses ofaviation accident statistics have found thatfrom 80 to 85 of accidents can be assignedbroadly to pilot error and the remainder tomechanical malfunctions To determine whypilots are making accident-causing errors itis useful to provide a more molecular exami-nation of pilot activities One such classifica-tion (Roscoe 1980) divides these activitiesinto three categories as follows

(I) ProceduralActivities-the management of thepower plant fuel vehicle configuration au-topilot displays navigation and communica-tion

(2) Perceptual-Motor Activities-including vehiclecontrol judgment of distance speed altitudeclearance hazard detection and geographicorientation

(3) Decision Activities-including the self-assessment of skill knowledge physical andpsychological capabilities the assessment ofaircraft and ground-system capabilitieshazard assessment navigation planningand flight priority adjustment

To determine the relative importance ofeach of these activities in civil aviation acci-dents statistics from the National Transpor-tation Safety Board (NTSB) Automated Air-craft Accident and Incident Information Sys-tem from 1970 through 1974 were classifiedinto the three behavioral categories givenabove (Jensen and Benel 1977) Then thetotal numbers of both fatal and nonfatal acci-dents during the 5-year period were deter-mined for each of these behavioral categoriesThe results of these analyses are shown inTable 1

Examinations of these data provide valu-

TABLE I

Number and Percentage of the Total GeneralAviation Accidents in Which the Pilot Is Listedas a Cause or Factor (1970-1974)

Fatal Nonfatal

Procedural 264 (46) 2 230 ( 86)Perceptual-Motor 2496 (438) 14561 (563)Decisional 2 940 (516) 9 087 (351)

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--63

able indications of possible weaknesses incurrent civil aviation programs For examplea majority of the nonfatal pilot-caused acci-dents (563) were the result of faultyperceptual-motor behavior The most signifi-cant factors here (failure to maintain flyingspeed and misjudgment of distance speedaltitude or clearance) represent one type ofpilot judgment On the other hand a majorityof the fatal pilot-caused accidents (516)were the result of faulty decision-making be-havior another type of pilot judgment Themost significant factors in this area were thefamiliar continued VFR (visual flight rules)into known adverse weather and inade-quate preflight planning or preparation

It is apparent from these accident statisticsthat both aspects of the decision-makingfunction are important to safe flight and maybe neglected in the present training and test-ing process However because it suffers fromgreater misunderstanding in aviation circlespilot judgment as represented by general de-cisional activities is the topic of concern inthis paper Although a significant amount ofresearch has been done on the decisional as-pect of judgment in recent years (Janis andMann 1977) no one has specifically exam-ined the judgment problem faced by the pilotthe flight instructor and the pilot examiner

Three major tasks must be accomplishedbefore major improvements in pilot trainingand evaluation can be realized The first taskis to establish a common definition of judg-ment as it applies to flying At present eventhough the term is used repeatedly in avia-tion circles and although FAAexaminers arerequired to evaluate candidates on the basisof judgment no such definition exists

The second major task is to determinewhether or not pilot judgment can be taughtand if so how best to teach it Since someaspects of pilot judgment are closely akin topersonality characteristics they may be dif-ficult to modify It might be necessary to use

testing and selection procedures to improvethis aspect of aviation safety and effective-ness Other aspects of pilot judgment aremore easily modified through systematictraining procedures

The third major task is to determinewhether or not pilot judgment can be evalu-ated reliably meaningfully and objectivelyBecause judgment is primarily a mental pro-cess it may be difficult to evaluate it reliablyOn the other hand behavioral events fre-quently have been used to indicate mental ac-tivity Although personality tests have provedto be somewhat unreliable research resultsusing these instruments may be useful in thedevelopment of instruments for evaluatingand predicting judgment-making behavior(Fishbein and Ajzen 1975)

Judgment Definition

As indicated above the word judgmenthas been used to describe two somewhat dif-ferent mental processes in aviation Perhapsits most common usage has been to describethe mental activity that takes place at theperceptual-motor level The other describesthe mental activity involved in choosing acourse of action from among several alterna-tives Obviously this second usage of theterm is similar to the first in that both involvemaking choices However there is a basicdifference

The first refers to highly learned perceptualresponses that must be made in a very shorttime in some cases continuously The secondrefers to cognitive decisions for which setprocedures either have not been establishedor may have been forgotten Flight instruc-tors have used various terms referring to thistype of judgment including headworkthinking ahead and staying ahead of theaircraft In a more general sense pilots havereferred to it as professionalism or command-ability In contrast to the perceptual-motorjudgments selection of alternatives usually

64--February 1982

occurs when more time is available to evalu-ate the situation a larger number of possiblecourses of action must be considered andthere is a greater degree of uncertainty con-cerning the existing situation and possibleoutcomes For these reasons cognitive judg-ments have been the source of greater misun-derstanding in pilot training and evaluation

These two aspects of judgment may be con-sidered as two ends of a continuum based oncognitive complexity and decision time Atone end of the continuum are the commonperceptual judgments of distance altitudespeed and clearance These perceptualjudgments are less complex in that they in-volve fewer pieces (frequently one) of fairlyaccurate information from which responsesare determined with highly learned motorbehavior They may require simple responsesbut frequently call for immediate controlmovement At the other end are what mightbe called cognitive judgments As describedabove these judgments are complex in thatthey usually involve a large number of rele-vant pieces of highly probabilistic informa-tion they usually require the specification ofand choice from among several alternativesand they are frequently affected by emotionsvalues and social pressures In addition cog-nitive judgments usually permit some delib-eration before a control response is requiredCandidate judgment definition Considering

these factors a candidate definition of cogni-tive judgment in flying airplanes is

(I) The ability to search for and establish the rel-evance of all available information regardinga situation to specify alternative courses ofaction and to determine expected outcomesfrom each alternative

(2)The motivation to choose and authoritativelyexecute a suitable course of action within thetime frame permitted by the situation where(a) Suitable is an alternative consistentwith societal norms (b) Action includes noaction some action or action to seek moreinformation

The first part of the definition refers to in-

HUMAN FACTORS

tellectual or discriminating abilities Thispart depends upon human capabilities tosense store retrieve and integrate informa-tion This function is what Van Dam (in Jen-sen and Benel 1977) calls the discriminatingability in professional pilots It is analogousto detectability Cd) in signal detection theoryIt is purely rational and could be statedmathematically If it were possible to sepa-rate this part of human judgment from thesecond part (which it is not) people wouldsolve problems in much the same way as acomputer does This is not to say that the de-cisions would be error free Probabilistic in-formation is used and the performance is de-pendent upon the amount type and accuracyof information stored as well as on inherent andlearned capabilities to process information

The second part of the definition refers tomotivational tendencies The emphasis inthis case is on the directional aspects of moti-vation rather than on intensity It says that apart of human judgment is based upon biasfactors (costs and payoffs) or tendencies touse less than rational information (defined bythe society) in choosing courses of action Thesociety would probably consider the use ofany information other than that required todefine the safety risk (eg conveniencemonetary gain gain in self-esteem adventureseeking etc) as less than rational This partof human judgment is analogous to the re-sponse criterion (m in signal detection theoryIt is what Van Dam (in Jensen and Benel1977) has called the response pattern of theprofessional pilot If properly developed thispart of human judgment would tend to haltthe use of information not directly related tothe safety of the flight and to direct the pilotsdecision toward the use of rational processes

JUDGMENT TRAINING

One of the most important questions to beaddressed after establishing the definition is

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--65

whether or not pilot judgment as definedcan be modified through training The pau-city of judgment-training guidelines in pilot-training and training-research literaturemight lead one to conclude that pilot judg-ment cannot be taught Literature and syllabicommonly used in flight-instructor coursescontain large sections on how to teach themotor skills of flying but offer very little onhow to teach pilot judgment (see Federal Avi-ation Administration 1977) The typicalprivate-pilot course offers a scattering ofjudgment-making instruction in the areas ofweather avoidance and power-plant emer-gencies but does not offer systematic judg-ment training

However there is evidence in aviation thatat least one form of judgment training ieassigning procedures for every conceivablesituation that might arise may be effectiveIn the military these are referred to asBoldface training procedures (ThorpeMartin Edwards and Eddowes 1976) Sim-ilar training procedures used by the airlineshave even been more successful than thoseused by the military (Gibson 1969 TransWorld Airlines 1969)

Looking outside the field of aviation onefinds other evidence indicating that judgmentmaking may be taught For example al-though signal detection theory (SDT) was notdesigned specifically to handle cognitivejudgments many of its methods can be usedto explain and perhaps even to modify pilotjudgment behavior SDT divides an indi-viduals decision behavior into two compo-nents representing his or her sensitivity (d)and response criterion or bias (f3roughlycorresponding to the two aspects of thejudgment definition previously stated

The sensitivity is affected both by the in-equality of the stimulus in the situation (sig-nal vs background noise) and the quality ofthe sensory apparatus of the observer In cog-nitive judgment this is the intellective com-

ponent It refers both to the strength of theevidence favoring the selection of one alterna-tive over another and to the knowledge andability of the pilot to use effective strategiesin gathering and processing the evidence

On the other hand the response criterionrepresen ts the level of excitation abovewhich the observer is willing to say signaIt is the amount of relevant information in thepresence of irrelevant information or noiseneeded to tip the decision one way or theother The criterion level (ie the amount ofrelevant information required) is influencedby motivation knowledge of the signalsprobability of occurrence and the costs andpayoffs attendant with a given response Incognitive judgment the response criterion isthe motivative component

The response criterion can be manipulatedthrough a wide range of values by adjustingprobabilities costs and payoffs (Birdsall1955) One can infer from the vast amount ofpsychophysical decision research that cogni-tive judgments can be modified in a similarway The flight instructor communicates de-cision biases attitudes risk tendencies con-sideration for passenger safety and pilotmotivation by example if not by design at alllevels of pilot training The method by whichthe instructor teaches these values is influ-enced by the instructors assignment of prob-abilities costs and payoffs to actions of thestudent

Although the SDT model assumes that thesensitivity component is quite stable for agiven individual there is a growing field ofresearch indicating that when dealing withthe intellective component of cognitivejudgment sensitivity can be modified as weiFor example attempts have been made todiscover the mental processes that are usedby expert judges such as stockbrokers live-stock judges and medical diagnosticians inmaking their decisions (Shanteau and Phelps1977 Slovic 1969 Anderson 1969 Hoffman

66--February 1982

Slovic and Rorer 1968) The thesis is that ifmodels of the mental processes used by theseexperts in decision making were availablethey could be used in training others to usesimilar processes In each of the areasstudied judgment training traditionally oc-curs over a fairly long apprenticeship pro-gram in which the trainee observes the ex-pert making decisions and learns by this ob-servation However as in aviation because ofthe complexity of the information used tomake decisions observation or even trial anderror are inefficient training methods

Research on the motivational aspect ofcognitive judgment also indicates that train-ing can have a beneficial effect The majorresearch efforts in this area are reported byJanis and Mann (1977) These authorsspeaking from a clinical perspective beginwith the assumption that psychological stressis a frequent cause of errors in decision mak-ing They say that stress arises from at leasttwo sources First the decision maker is con-cerned about the material and social lossesthat may be suffered from whichever courseof action is chosen including the costs offailing to live up to prior commitments Sec-ond the pilots reputation and self-esteem asa competent decision maker are at stake Themore severe the anticipated losses thegreater the stress

Janis and Mann have constructed aconflict-theory model of decision makingpostulating that the way difficult choices areresolved is determined by the presence orabsence of three conditions awareness ofrisks involved hope of finding a better so-lution and time available in which to makethe decision They have developed severalclinical procedures to improve decisionmaking under the titles awareness of ra-tionalizations emotional role playingbalance sheet and outcome psychodra-ma They report that these procedures havedemonstrated effectiveness in changing

HUMAN FACTORS

decision-making tendencies and in modifyingattitudes

Some Learning Principles

Because some well-established learningprinciples are misapplied in many trainingprograms a discussion of these principles asapplied to pilot judgment training is neededPerhaps the most popular principle is thatthe best way to learn an activity is to practicethat activity (Gagne 1962) The assumption isrooted in much of the educational literatureand is often identified by the catch-phraselearning by doing Gagne points out that itmay also be a generalization from research onthe conditioned response in which learningparticularly in animals appears to have oc-curred only after a response (practice) hasbeen made

However Gagne argues that practice is notan effective training method by itself even forthe acquisition of such motor skills as fieldgunnery He says that instruction about thecorrect sighting picture for ranging is moreeffective in bringing about improved perfor-mance than is practice on the task (Gagne1962 p 85) The point is that training shouldemphasize the principles and procedures(thought processes) involved and practiceshould be directed to take advantage of theseprinciples If this is the proper emphasis forteaching motor skills it is even more impor-tant in the teaching of judgment-makingskills which are more deeply rooted inthought processes

A second learning principle that is fre-quently misapplied in training situations isvariously called reinforcement feedback orknowledge of results during practice Thisprinciple has been found to be most effectivein choice behavior However Gagne pointsout that some manipulations that artificiallyimprove feedback during practice failed toshow reliably better transfer to the opera-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-67

tional environment and that others showednegative transfer perhaps because of alearned dependence on the artificial cuesApparently the form ofthe feedback is important

Any beginning flight student will say thatsuch usual feedback information as you didit right or you did it wrong is almost use-less The time interval between execution andfeedback may be long the feedback is oftencluttered with interfering information andthe trials themselves are often so complexthat the student may learn very little fromsuch a response by the flight instructor Astudent really needs to know why the re-sponse was right or wrong He or she needs toknow what rules should have been followedand if and how those rules were violated Al-though practice and rightwrong types offeedback may be useful in some training situ-ations they should be deemphasized in favorof these thought-oriented teaching principlesin all types of pilot training but especially injudgment training

A Systematic Approach to Training

The need for pilot judgment training hasbeen established for all levels of flight in-struction Without a systematic judgment-training program good pilot judgment is ac-quired by the cautions and the lucky overyears of flying experience in many variedsituations The primary task of aviationeducators using systematic judgment-train-ing techniques is to compress a lifetime offlying experience into a relatively short train-ing program to instill good pilot judgmentinto the emerging pilot

In addition a ground school might includeinstruction on information integration andsubjective probability estimation (Goldberg1968) In expert judges judgment-makingbehavior is characterized by chunking or theformation of clusters of stimulus attributesand response alternatives for economy in the

thought process Ground school studentscould be taught to use these processes in theirjudgment-making activity The instructorwould show how various types of probabilis-tic information such as weather forecastspredicted aircraft system malfunctions andpredicted air traffic control problems shouldbe combined when making flight decisions

The instructor could teach the student howto anticipate decisions that might have to bemade later as a result of immediate choicesof action Such anticipation permits thegathering of relevant information under lowerlevels of stress when errors are less frequentrather than later in the flight when theamount of time available to decide may be-come an error-causing factor This section ofground school could also include decision-making training using procedures suggestedby Janis and Mann (1977) such as balancesheet and emotional role playingComputer-assisted instruction An instruc-

tion technique that holds exceptional prom-ise for pilot judgment training and evalua-tion is computer-assisted instruction (CAl)The great advantage of CAl systems is thatthey can be used to teach principles and thento permit the student to participate indecision-making processes a highly effectivelearning technique (Fishbein and Ajzen1975) The disadvantage of these systems inthe past has been their limited availabilityand high cost However recent advances intechnology are making them available at arelatively low cost (Trollip and Ortony 1977)

Although CAl programs are available inseveral forms the dialogue systems that per-mit student-computer interaction unre-stricted by preset response alternatives (Al-pert and Bitzer 1970) show the greatestpotential for application to instruction injudgment making These systems dependupon a set of stored algorithms that are usedby the computer to construct a great varietyof responses to student questions In addition

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

62--February 1982

the book to determine the expected flightperformance Finally the present and fore-cast weather the terrain and the expectedtraffic density must be checked These arethen compared with the pilots estimate ofher or his own capability before determiningwhether or not the planned flight will be safe

Examples of decisions being made withimperfect information can be found in allareas of flight activity Furthermore everydecision that the pilot makes is colored byphysiological psychological and social pres-sures that are virtually impossible to weighproperly on the spot For example just aspersons watching a sporting event may seean infraction or foul differently dependingupon their vantage points and which teamthey support a pilot may be influenced toview the weather outlook or his or her ownabilities differently depending upon the im-portance or value he or she assigns to a givenflight Evidence from the social psychologyliterature (Kogan and Wallach 1964)suggests that ones self-image and the need tomaintain an external image largely deter-mine how much effect the costs and payoffsfor a given flight will have on the judgment ofones own ability to make a safe flight

Some pilots may be susceptible to socialpressures that result in less than rationalpilot judgment Irrational pilot judgment hasbeen exhibited by such unsafe practices asflying under bridges landing on busy high-ways attempting to land in football stadi-ums and flying formation on other un-suspecting pilots Potential sources of so-cial pressure that may lead to these types ofactivities include peer reactions fear of fail-ure censure from superiors or family mem-bers and many other factors (Janis andMann 1977)

The Problem

An analysis of accident statistics bycategories of pilot activities reveals the seri-

HUMAN FACTORS

ous nature of the problem Most analyses ofaviation accident statistics have found thatfrom 80 to 85 of accidents can be assignedbroadly to pilot error and the remainder tomechanical malfunctions To determine whypilots are making accident-causing errors itis useful to provide a more molecular exami-nation of pilot activities One such classifica-tion (Roscoe 1980) divides these activitiesinto three categories as follows

(I) ProceduralActivities-the management of thepower plant fuel vehicle configuration au-topilot displays navigation and communica-tion

(2) Perceptual-Motor Activities-including vehiclecontrol judgment of distance speed altitudeclearance hazard detection and geographicorientation

(3) Decision Activities-including the self-assessment of skill knowledge physical andpsychological capabilities the assessment ofaircraft and ground-system capabilitieshazard assessment navigation planningand flight priority adjustment

To determine the relative importance ofeach of these activities in civil aviation acci-dents statistics from the National Transpor-tation Safety Board (NTSB) Automated Air-craft Accident and Incident Information Sys-tem from 1970 through 1974 were classifiedinto the three behavioral categories givenabove (Jensen and Benel 1977) Then thetotal numbers of both fatal and nonfatal acci-dents during the 5-year period were deter-mined for each of these behavioral categoriesThe results of these analyses are shown inTable 1

Examinations of these data provide valu-

TABLE I

Number and Percentage of the Total GeneralAviation Accidents in Which the Pilot Is Listedas a Cause or Factor (1970-1974)

Fatal Nonfatal

Procedural 264 (46) 2 230 ( 86)Perceptual-Motor 2496 (438) 14561 (563)Decisional 2 940 (516) 9 087 (351)

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--63

able indications of possible weaknesses incurrent civil aviation programs For examplea majority of the nonfatal pilot-caused acci-dents (563) were the result of faultyperceptual-motor behavior The most signifi-cant factors here (failure to maintain flyingspeed and misjudgment of distance speedaltitude or clearance) represent one type ofpilot judgment On the other hand a majorityof the fatal pilot-caused accidents (516)were the result of faulty decision-making be-havior another type of pilot judgment Themost significant factors in this area were thefamiliar continued VFR (visual flight rules)into known adverse weather and inade-quate preflight planning or preparation

It is apparent from these accident statisticsthat both aspects of the decision-makingfunction are important to safe flight and maybe neglected in the present training and test-ing process However because it suffers fromgreater misunderstanding in aviation circlespilot judgment as represented by general de-cisional activities is the topic of concern inthis paper Although a significant amount ofresearch has been done on the decisional as-pect of judgment in recent years (Janis andMann 1977) no one has specifically exam-ined the judgment problem faced by the pilotthe flight instructor and the pilot examiner

Three major tasks must be accomplishedbefore major improvements in pilot trainingand evaluation can be realized The first taskis to establish a common definition of judg-ment as it applies to flying At present eventhough the term is used repeatedly in avia-tion circles and although FAAexaminers arerequired to evaluate candidates on the basisof judgment no such definition exists

The second major task is to determinewhether or not pilot judgment can be taughtand if so how best to teach it Since someaspects of pilot judgment are closely akin topersonality characteristics they may be dif-ficult to modify It might be necessary to use

testing and selection procedures to improvethis aspect of aviation safety and effective-ness Other aspects of pilot judgment aremore easily modified through systematictraining procedures

The third major task is to determinewhether or not pilot judgment can be evalu-ated reliably meaningfully and objectivelyBecause judgment is primarily a mental pro-cess it may be difficult to evaluate it reliablyOn the other hand behavioral events fre-quently have been used to indicate mental ac-tivity Although personality tests have provedto be somewhat unreliable research resultsusing these instruments may be useful in thedevelopment of instruments for evaluatingand predicting judgment-making behavior(Fishbein and Ajzen 1975)

Judgment Definition

As indicated above the word judgmenthas been used to describe two somewhat dif-ferent mental processes in aviation Perhapsits most common usage has been to describethe mental activity that takes place at theperceptual-motor level The other describesthe mental activity involved in choosing acourse of action from among several alterna-tives Obviously this second usage of theterm is similar to the first in that both involvemaking choices However there is a basicdifference

The first refers to highly learned perceptualresponses that must be made in a very shorttime in some cases continuously The secondrefers to cognitive decisions for which setprocedures either have not been establishedor may have been forgotten Flight instruc-tors have used various terms referring to thistype of judgment including headworkthinking ahead and staying ahead of theaircraft In a more general sense pilots havereferred to it as professionalism or command-ability In contrast to the perceptual-motorjudgments selection of alternatives usually

64--February 1982

occurs when more time is available to evalu-ate the situation a larger number of possiblecourses of action must be considered andthere is a greater degree of uncertainty con-cerning the existing situation and possibleoutcomes For these reasons cognitive judg-ments have been the source of greater misun-derstanding in pilot training and evaluation

These two aspects of judgment may be con-sidered as two ends of a continuum based oncognitive complexity and decision time Atone end of the continuum are the commonperceptual judgments of distance altitudespeed and clearance These perceptualjudgments are less complex in that they in-volve fewer pieces (frequently one) of fairlyaccurate information from which responsesare determined with highly learned motorbehavior They may require simple responsesbut frequently call for immediate controlmovement At the other end are what mightbe called cognitive judgments As describedabove these judgments are complex in thatthey usually involve a large number of rele-vant pieces of highly probabilistic informa-tion they usually require the specification ofand choice from among several alternativesand they are frequently affected by emotionsvalues and social pressures In addition cog-nitive judgments usually permit some delib-eration before a control response is requiredCandidate judgment definition Considering

these factors a candidate definition of cogni-tive judgment in flying airplanes is

(I) The ability to search for and establish the rel-evance of all available information regardinga situation to specify alternative courses ofaction and to determine expected outcomesfrom each alternative

(2)The motivation to choose and authoritativelyexecute a suitable course of action within thetime frame permitted by the situation where(a) Suitable is an alternative consistentwith societal norms (b) Action includes noaction some action or action to seek moreinformation

The first part of the definition refers to in-

HUMAN FACTORS

tellectual or discriminating abilities Thispart depends upon human capabilities tosense store retrieve and integrate informa-tion This function is what Van Dam (in Jen-sen and Benel 1977) calls the discriminatingability in professional pilots It is analogousto detectability Cd) in signal detection theoryIt is purely rational and could be statedmathematically If it were possible to sepa-rate this part of human judgment from thesecond part (which it is not) people wouldsolve problems in much the same way as acomputer does This is not to say that the de-cisions would be error free Probabilistic in-formation is used and the performance is de-pendent upon the amount type and accuracyof information stored as well as on inherent andlearned capabilities to process information

The second part of the definition refers tomotivational tendencies The emphasis inthis case is on the directional aspects of moti-vation rather than on intensity It says that apart of human judgment is based upon biasfactors (costs and payoffs) or tendencies touse less than rational information (defined bythe society) in choosing courses of action Thesociety would probably consider the use ofany information other than that required todefine the safety risk (eg conveniencemonetary gain gain in self-esteem adventureseeking etc) as less than rational This partof human judgment is analogous to the re-sponse criterion (m in signal detection theoryIt is what Van Dam (in Jensen and Benel1977) has called the response pattern of theprofessional pilot If properly developed thispart of human judgment would tend to haltthe use of information not directly related tothe safety of the flight and to direct the pilotsdecision toward the use of rational processes

JUDGMENT TRAINING

One of the most important questions to beaddressed after establishing the definition is

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--65

whether or not pilot judgment as definedcan be modified through training The pau-city of judgment-training guidelines in pilot-training and training-research literaturemight lead one to conclude that pilot judg-ment cannot be taught Literature and syllabicommonly used in flight-instructor coursescontain large sections on how to teach themotor skills of flying but offer very little onhow to teach pilot judgment (see Federal Avi-ation Administration 1977) The typicalprivate-pilot course offers a scattering ofjudgment-making instruction in the areas ofweather avoidance and power-plant emer-gencies but does not offer systematic judg-ment training

However there is evidence in aviation thatat least one form of judgment training ieassigning procedures for every conceivablesituation that might arise may be effectiveIn the military these are referred to asBoldface training procedures (ThorpeMartin Edwards and Eddowes 1976) Sim-ilar training procedures used by the airlineshave even been more successful than thoseused by the military (Gibson 1969 TransWorld Airlines 1969)

Looking outside the field of aviation onefinds other evidence indicating that judgmentmaking may be taught For example al-though signal detection theory (SDT) was notdesigned specifically to handle cognitivejudgments many of its methods can be usedto explain and perhaps even to modify pilotjudgment behavior SDT divides an indi-viduals decision behavior into two compo-nents representing his or her sensitivity (d)and response criterion or bias (f3roughlycorresponding to the two aspects of thejudgment definition previously stated

The sensitivity is affected both by the in-equality of the stimulus in the situation (sig-nal vs background noise) and the quality ofthe sensory apparatus of the observer In cog-nitive judgment this is the intellective com-

ponent It refers both to the strength of theevidence favoring the selection of one alterna-tive over another and to the knowledge andability of the pilot to use effective strategiesin gathering and processing the evidence

On the other hand the response criterionrepresen ts the level of excitation abovewhich the observer is willing to say signaIt is the amount of relevant information in thepresence of irrelevant information or noiseneeded to tip the decision one way or theother The criterion level (ie the amount ofrelevant information required) is influencedby motivation knowledge of the signalsprobability of occurrence and the costs andpayoffs attendant with a given response Incognitive judgment the response criterion isthe motivative component

The response criterion can be manipulatedthrough a wide range of values by adjustingprobabilities costs and payoffs (Birdsall1955) One can infer from the vast amount ofpsychophysical decision research that cogni-tive judgments can be modified in a similarway The flight instructor communicates de-cision biases attitudes risk tendencies con-sideration for passenger safety and pilotmotivation by example if not by design at alllevels of pilot training The method by whichthe instructor teaches these values is influ-enced by the instructors assignment of prob-abilities costs and payoffs to actions of thestudent

Although the SDT model assumes that thesensitivity component is quite stable for agiven individual there is a growing field ofresearch indicating that when dealing withthe intellective component of cognitivejudgment sensitivity can be modified as weiFor example attempts have been made todiscover the mental processes that are usedby expert judges such as stockbrokers live-stock judges and medical diagnosticians inmaking their decisions (Shanteau and Phelps1977 Slovic 1969 Anderson 1969 Hoffman

66--February 1982

Slovic and Rorer 1968) The thesis is that ifmodels of the mental processes used by theseexperts in decision making were availablethey could be used in training others to usesimilar processes In each of the areasstudied judgment training traditionally oc-curs over a fairly long apprenticeship pro-gram in which the trainee observes the ex-pert making decisions and learns by this ob-servation However as in aviation because ofthe complexity of the information used tomake decisions observation or even trial anderror are inefficient training methods

Research on the motivational aspect ofcognitive judgment also indicates that train-ing can have a beneficial effect The majorresearch efforts in this area are reported byJanis and Mann (1977) These authorsspeaking from a clinical perspective beginwith the assumption that psychological stressis a frequent cause of errors in decision mak-ing They say that stress arises from at leasttwo sources First the decision maker is con-cerned about the material and social lossesthat may be suffered from whichever courseof action is chosen including the costs offailing to live up to prior commitments Sec-ond the pilots reputation and self-esteem asa competent decision maker are at stake Themore severe the anticipated losses thegreater the stress

Janis and Mann have constructed aconflict-theory model of decision makingpostulating that the way difficult choices areresolved is determined by the presence orabsence of three conditions awareness ofrisks involved hope of finding a better so-lution and time available in which to makethe decision They have developed severalclinical procedures to improve decisionmaking under the titles awareness of ra-tionalizations emotional role playingbalance sheet and outcome psychodra-ma They report that these procedures havedemonstrated effectiveness in changing

HUMAN FACTORS

decision-making tendencies and in modifyingattitudes

Some Learning Principles

Because some well-established learningprinciples are misapplied in many trainingprograms a discussion of these principles asapplied to pilot judgment training is neededPerhaps the most popular principle is thatthe best way to learn an activity is to practicethat activity (Gagne 1962) The assumption isrooted in much of the educational literatureand is often identified by the catch-phraselearning by doing Gagne points out that itmay also be a generalization from research onthe conditioned response in which learningparticularly in animals appears to have oc-curred only after a response (practice) hasbeen made

However Gagne argues that practice is notan effective training method by itself even forthe acquisition of such motor skills as fieldgunnery He says that instruction about thecorrect sighting picture for ranging is moreeffective in bringing about improved perfor-mance than is practice on the task (Gagne1962 p 85) The point is that training shouldemphasize the principles and procedures(thought processes) involved and practiceshould be directed to take advantage of theseprinciples If this is the proper emphasis forteaching motor skills it is even more impor-tant in the teaching of judgment-makingskills which are more deeply rooted inthought processes

A second learning principle that is fre-quently misapplied in training situations isvariously called reinforcement feedback orknowledge of results during practice Thisprinciple has been found to be most effectivein choice behavior However Gagne pointsout that some manipulations that artificiallyimprove feedback during practice failed toshow reliably better transfer to the opera-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-67

tional environment and that others showednegative transfer perhaps because of alearned dependence on the artificial cuesApparently the form ofthe feedback is important

Any beginning flight student will say thatsuch usual feedback information as you didit right or you did it wrong is almost use-less The time interval between execution andfeedback may be long the feedback is oftencluttered with interfering information andthe trials themselves are often so complexthat the student may learn very little fromsuch a response by the flight instructor Astudent really needs to know why the re-sponse was right or wrong He or she needs toknow what rules should have been followedand if and how those rules were violated Al-though practice and rightwrong types offeedback may be useful in some training situ-ations they should be deemphasized in favorof these thought-oriented teaching principlesin all types of pilot training but especially injudgment training

A Systematic Approach to Training

The need for pilot judgment training hasbeen established for all levels of flight in-struction Without a systematic judgment-training program good pilot judgment is ac-quired by the cautions and the lucky overyears of flying experience in many variedsituations The primary task of aviationeducators using systematic judgment-train-ing techniques is to compress a lifetime offlying experience into a relatively short train-ing program to instill good pilot judgmentinto the emerging pilot

In addition a ground school might includeinstruction on information integration andsubjective probability estimation (Goldberg1968) In expert judges judgment-makingbehavior is characterized by chunking or theformation of clusters of stimulus attributesand response alternatives for economy in the

thought process Ground school studentscould be taught to use these processes in theirjudgment-making activity The instructorwould show how various types of probabilis-tic information such as weather forecastspredicted aircraft system malfunctions andpredicted air traffic control problems shouldbe combined when making flight decisions

The instructor could teach the student howto anticipate decisions that might have to bemade later as a result of immediate choicesof action Such anticipation permits thegathering of relevant information under lowerlevels of stress when errors are less frequentrather than later in the flight when theamount of time available to decide may be-come an error-causing factor This section ofground school could also include decision-making training using procedures suggestedby Janis and Mann (1977) such as balancesheet and emotional role playingComputer-assisted instruction An instruc-

tion technique that holds exceptional prom-ise for pilot judgment training and evalua-tion is computer-assisted instruction (CAl)The great advantage of CAl systems is thatthey can be used to teach principles and thento permit the student to participate indecision-making processes a highly effectivelearning technique (Fishbein and Ajzen1975) The disadvantage of these systems inthe past has been their limited availabilityand high cost However recent advances intechnology are making them available at arelatively low cost (Trollip and Ortony 1977)

Although CAl programs are available inseveral forms the dialogue systems that per-mit student-computer interaction unre-stricted by preset response alternatives (Al-pert and Bitzer 1970) show the greatestpotential for application to instruction injudgment making These systems dependupon a set of stored algorithms that are usedby the computer to construct a great varietyof responses to student questions In addition

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--63

able indications of possible weaknesses incurrent civil aviation programs For examplea majority of the nonfatal pilot-caused acci-dents (563) were the result of faultyperceptual-motor behavior The most signifi-cant factors here (failure to maintain flyingspeed and misjudgment of distance speedaltitude or clearance) represent one type ofpilot judgment On the other hand a majorityof the fatal pilot-caused accidents (516)were the result of faulty decision-making be-havior another type of pilot judgment Themost significant factors in this area were thefamiliar continued VFR (visual flight rules)into known adverse weather and inade-quate preflight planning or preparation

It is apparent from these accident statisticsthat both aspects of the decision-makingfunction are important to safe flight and maybe neglected in the present training and test-ing process However because it suffers fromgreater misunderstanding in aviation circlespilot judgment as represented by general de-cisional activities is the topic of concern inthis paper Although a significant amount ofresearch has been done on the decisional as-pect of judgment in recent years (Janis andMann 1977) no one has specifically exam-ined the judgment problem faced by the pilotthe flight instructor and the pilot examiner

Three major tasks must be accomplishedbefore major improvements in pilot trainingand evaluation can be realized The first taskis to establish a common definition of judg-ment as it applies to flying At present eventhough the term is used repeatedly in avia-tion circles and although FAAexaminers arerequired to evaluate candidates on the basisof judgment no such definition exists

The second major task is to determinewhether or not pilot judgment can be taughtand if so how best to teach it Since someaspects of pilot judgment are closely akin topersonality characteristics they may be dif-ficult to modify It might be necessary to use

testing and selection procedures to improvethis aspect of aviation safety and effective-ness Other aspects of pilot judgment aremore easily modified through systematictraining procedures

The third major task is to determinewhether or not pilot judgment can be evalu-ated reliably meaningfully and objectivelyBecause judgment is primarily a mental pro-cess it may be difficult to evaluate it reliablyOn the other hand behavioral events fre-quently have been used to indicate mental ac-tivity Although personality tests have provedto be somewhat unreliable research resultsusing these instruments may be useful in thedevelopment of instruments for evaluatingand predicting judgment-making behavior(Fishbein and Ajzen 1975)

Judgment Definition

As indicated above the word judgmenthas been used to describe two somewhat dif-ferent mental processes in aviation Perhapsits most common usage has been to describethe mental activity that takes place at theperceptual-motor level The other describesthe mental activity involved in choosing acourse of action from among several alterna-tives Obviously this second usage of theterm is similar to the first in that both involvemaking choices However there is a basicdifference

The first refers to highly learned perceptualresponses that must be made in a very shorttime in some cases continuously The secondrefers to cognitive decisions for which setprocedures either have not been establishedor may have been forgotten Flight instruc-tors have used various terms referring to thistype of judgment including headworkthinking ahead and staying ahead of theaircraft In a more general sense pilots havereferred to it as professionalism or command-ability In contrast to the perceptual-motorjudgments selection of alternatives usually

64--February 1982

occurs when more time is available to evalu-ate the situation a larger number of possiblecourses of action must be considered andthere is a greater degree of uncertainty con-cerning the existing situation and possibleoutcomes For these reasons cognitive judg-ments have been the source of greater misun-derstanding in pilot training and evaluation

These two aspects of judgment may be con-sidered as two ends of a continuum based oncognitive complexity and decision time Atone end of the continuum are the commonperceptual judgments of distance altitudespeed and clearance These perceptualjudgments are less complex in that they in-volve fewer pieces (frequently one) of fairlyaccurate information from which responsesare determined with highly learned motorbehavior They may require simple responsesbut frequently call for immediate controlmovement At the other end are what mightbe called cognitive judgments As describedabove these judgments are complex in thatthey usually involve a large number of rele-vant pieces of highly probabilistic informa-tion they usually require the specification ofand choice from among several alternativesand they are frequently affected by emotionsvalues and social pressures In addition cog-nitive judgments usually permit some delib-eration before a control response is requiredCandidate judgment definition Considering

these factors a candidate definition of cogni-tive judgment in flying airplanes is

(I) The ability to search for and establish the rel-evance of all available information regardinga situation to specify alternative courses ofaction and to determine expected outcomesfrom each alternative

(2)The motivation to choose and authoritativelyexecute a suitable course of action within thetime frame permitted by the situation where(a) Suitable is an alternative consistentwith societal norms (b) Action includes noaction some action or action to seek moreinformation

The first part of the definition refers to in-

HUMAN FACTORS

tellectual or discriminating abilities Thispart depends upon human capabilities tosense store retrieve and integrate informa-tion This function is what Van Dam (in Jen-sen and Benel 1977) calls the discriminatingability in professional pilots It is analogousto detectability Cd) in signal detection theoryIt is purely rational and could be statedmathematically If it were possible to sepa-rate this part of human judgment from thesecond part (which it is not) people wouldsolve problems in much the same way as acomputer does This is not to say that the de-cisions would be error free Probabilistic in-formation is used and the performance is de-pendent upon the amount type and accuracyof information stored as well as on inherent andlearned capabilities to process information

The second part of the definition refers tomotivational tendencies The emphasis inthis case is on the directional aspects of moti-vation rather than on intensity It says that apart of human judgment is based upon biasfactors (costs and payoffs) or tendencies touse less than rational information (defined bythe society) in choosing courses of action Thesociety would probably consider the use ofany information other than that required todefine the safety risk (eg conveniencemonetary gain gain in self-esteem adventureseeking etc) as less than rational This partof human judgment is analogous to the re-sponse criterion (m in signal detection theoryIt is what Van Dam (in Jensen and Benel1977) has called the response pattern of theprofessional pilot If properly developed thispart of human judgment would tend to haltthe use of information not directly related tothe safety of the flight and to direct the pilotsdecision toward the use of rational processes

JUDGMENT TRAINING

One of the most important questions to beaddressed after establishing the definition is

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--65

whether or not pilot judgment as definedcan be modified through training The pau-city of judgment-training guidelines in pilot-training and training-research literaturemight lead one to conclude that pilot judg-ment cannot be taught Literature and syllabicommonly used in flight-instructor coursescontain large sections on how to teach themotor skills of flying but offer very little onhow to teach pilot judgment (see Federal Avi-ation Administration 1977) The typicalprivate-pilot course offers a scattering ofjudgment-making instruction in the areas ofweather avoidance and power-plant emer-gencies but does not offer systematic judg-ment training

However there is evidence in aviation thatat least one form of judgment training ieassigning procedures for every conceivablesituation that might arise may be effectiveIn the military these are referred to asBoldface training procedures (ThorpeMartin Edwards and Eddowes 1976) Sim-ilar training procedures used by the airlineshave even been more successful than thoseused by the military (Gibson 1969 TransWorld Airlines 1969)

Looking outside the field of aviation onefinds other evidence indicating that judgmentmaking may be taught For example al-though signal detection theory (SDT) was notdesigned specifically to handle cognitivejudgments many of its methods can be usedto explain and perhaps even to modify pilotjudgment behavior SDT divides an indi-viduals decision behavior into two compo-nents representing his or her sensitivity (d)and response criterion or bias (f3roughlycorresponding to the two aspects of thejudgment definition previously stated

The sensitivity is affected both by the in-equality of the stimulus in the situation (sig-nal vs background noise) and the quality ofthe sensory apparatus of the observer In cog-nitive judgment this is the intellective com-

ponent It refers both to the strength of theevidence favoring the selection of one alterna-tive over another and to the knowledge andability of the pilot to use effective strategiesin gathering and processing the evidence

On the other hand the response criterionrepresen ts the level of excitation abovewhich the observer is willing to say signaIt is the amount of relevant information in thepresence of irrelevant information or noiseneeded to tip the decision one way or theother The criterion level (ie the amount ofrelevant information required) is influencedby motivation knowledge of the signalsprobability of occurrence and the costs andpayoffs attendant with a given response Incognitive judgment the response criterion isthe motivative component

The response criterion can be manipulatedthrough a wide range of values by adjustingprobabilities costs and payoffs (Birdsall1955) One can infer from the vast amount ofpsychophysical decision research that cogni-tive judgments can be modified in a similarway The flight instructor communicates de-cision biases attitudes risk tendencies con-sideration for passenger safety and pilotmotivation by example if not by design at alllevels of pilot training The method by whichthe instructor teaches these values is influ-enced by the instructors assignment of prob-abilities costs and payoffs to actions of thestudent

Although the SDT model assumes that thesensitivity component is quite stable for agiven individual there is a growing field ofresearch indicating that when dealing withthe intellective component of cognitivejudgment sensitivity can be modified as weiFor example attempts have been made todiscover the mental processes that are usedby expert judges such as stockbrokers live-stock judges and medical diagnosticians inmaking their decisions (Shanteau and Phelps1977 Slovic 1969 Anderson 1969 Hoffman

66--February 1982

Slovic and Rorer 1968) The thesis is that ifmodels of the mental processes used by theseexperts in decision making were availablethey could be used in training others to usesimilar processes In each of the areasstudied judgment training traditionally oc-curs over a fairly long apprenticeship pro-gram in which the trainee observes the ex-pert making decisions and learns by this ob-servation However as in aviation because ofthe complexity of the information used tomake decisions observation or even trial anderror are inefficient training methods

Research on the motivational aspect ofcognitive judgment also indicates that train-ing can have a beneficial effect The majorresearch efforts in this area are reported byJanis and Mann (1977) These authorsspeaking from a clinical perspective beginwith the assumption that psychological stressis a frequent cause of errors in decision mak-ing They say that stress arises from at leasttwo sources First the decision maker is con-cerned about the material and social lossesthat may be suffered from whichever courseof action is chosen including the costs offailing to live up to prior commitments Sec-ond the pilots reputation and self-esteem asa competent decision maker are at stake Themore severe the anticipated losses thegreater the stress

Janis and Mann have constructed aconflict-theory model of decision makingpostulating that the way difficult choices areresolved is determined by the presence orabsence of three conditions awareness ofrisks involved hope of finding a better so-lution and time available in which to makethe decision They have developed severalclinical procedures to improve decisionmaking under the titles awareness of ra-tionalizations emotional role playingbalance sheet and outcome psychodra-ma They report that these procedures havedemonstrated effectiveness in changing

HUMAN FACTORS

decision-making tendencies and in modifyingattitudes

Some Learning Principles

Because some well-established learningprinciples are misapplied in many trainingprograms a discussion of these principles asapplied to pilot judgment training is neededPerhaps the most popular principle is thatthe best way to learn an activity is to practicethat activity (Gagne 1962) The assumption isrooted in much of the educational literatureand is often identified by the catch-phraselearning by doing Gagne points out that itmay also be a generalization from research onthe conditioned response in which learningparticularly in animals appears to have oc-curred only after a response (practice) hasbeen made

However Gagne argues that practice is notan effective training method by itself even forthe acquisition of such motor skills as fieldgunnery He says that instruction about thecorrect sighting picture for ranging is moreeffective in bringing about improved perfor-mance than is practice on the task (Gagne1962 p 85) The point is that training shouldemphasize the principles and procedures(thought processes) involved and practiceshould be directed to take advantage of theseprinciples If this is the proper emphasis forteaching motor skills it is even more impor-tant in the teaching of judgment-makingskills which are more deeply rooted inthought processes

A second learning principle that is fre-quently misapplied in training situations isvariously called reinforcement feedback orknowledge of results during practice Thisprinciple has been found to be most effectivein choice behavior However Gagne pointsout that some manipulations that artificiallyimprove feedback during practice failed toshow reliably better transfer to the opera-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-67

tional environment and that others showednegative transfer perhaps because of alearned dependence on the artificial cuesApparently the form ofthe feedback is important

Any beginning flight student will say thatsuch usual feedback information as you didit right or you did it wrong is almost use-less The time interval between execution andfeedback may be long the feedback is oftencluttered with interfering information andthe trials themselves are often so complexthat the student may learn very little fromsuch a response by the flight instructor Astudent really needs to know why the re-sponse was right or wrong He or she needs toknow what rules should have been followedand if and how those rules were violated Al-though practice and rightwrong types offeedback may be useful in some training situ-ations they should be deemphasized in favorof these thought-oriented teaching principlesin all types of pilot training but especially injudgment training

A Systematic Approach to Training

The need for pilot judgment training hasbeen established for all levels of flight in-struction Without a systematic judgment-training program good pilot judgment is ac-quired by the cautions and the lucky overyears of flying experience in many variedsituations The primary task of aviationeducators using systematic judgment-train-ing techniques is to compress a lifetime offlying experience into a relatively short train-ing program to instill good pilot judgmentinto the emerging pilot

In addition a ground school might includeinstruction on information integration andsubjective probability estimation (Goldberg1968) In expert judges judgment-makingbehavior is characterized by chunking or theformation of clusters of stimulus attributesand response alternatives for economy in the

thought process Ground school studentscould be taught to use these processes in theirjudgment-making activity The instructorwould show how various types of probabilis-tic information such as weather forecastspredicted aircraft system malfunctions andpredicted air traffic control problems shouldbe combined when making flight decisions

The instructor could teach the student howto anticipate decisions that might have to bemade later as a result of immediate choicesof action Such anticipation permits thegathering of relevant information under lowerlevels of stress when errors are less frequentrather than later in the flight when theamount of time available to decide may be-come an error-causing factor This section ofground school could also include decision-making training using procedures suggestedby Janis and Mann (1977) such as balancesheet and emotional role playingComputer-assisted instruction An instruc-

tion technique that holds exceptional prom-ise for pilot judgment training and evalua-tion is computer-assisted instruction (CAl)The great advantage of CAl systems is thatthey can be used to teach principles and thento permit the student to participate indecision-making processes a highly effectivelearning technique (Fishbein and Ajzen1975) The disadvantage of these systems inthe past has been their limited availabilityand high cost However recent advances intechnology are making them available at arelatively low cost (Trollip and Ortony 1977)

Although CAl programs are available inseveral forms the dialogue systems that per-mit student-computer interaction unre-stricted by preset response alternatives (Al-pert and Bitzer 1970) show the greatestpotential for application to instruction injudgment making These systems dependupon a set of stored algorithms that are usedby the computer to construct a great varietyof responses to student questions In addition

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

64--February 1982

occurs when more time is available to evalu-ate the situation a larger number of possiblecourses of action must be considered andthere is a greater degree of uncertainty con-cerning the existing situation and possibleoutcomes For these reasons cognitive judg-ments have been the source of greater misun-derstanding in pilot training and evaluation

These two aspects of judgment may be con-sidered as two ends of a continuum based oncognitive complexity and decision time Atone end of the continuum are the commonperceptual judgments of distance altitudespeed and clearance These perceptualjudgments are less complex in that they in-volve fewer pieces (frequently one) of fairlyaccurate information from which responsesare determined with highly learned motorbehavior They may require simple responsesbut frequently call for immediate controlmovement At the other end are what mightbe called cognitive judgments As describedabove these judgments are complex in thatthey usually involve a large number of rele-vant pieces of highly probabilistic informa-tion they usually require the specification ofand choice from among several alternativesand they are frequently affected by emotionsvalues and social pressures In addition cog-nitive judgments usually permit some delib-eration before a control response is requiredCandidate judgment definition Considering

these factors a candidate definition of cogni-tive judgment in flying airplanes is

(I) The ability to search for and establish the rel-evance of all available information regardinga situation to specify alternative courses ofaction and to determine expected outcomesfrom each alternative

(2)The motivation to choose and authoritativelyexecute a suitable course of action within thetime frame permitted by the situation where(a) Suitable is an alternative consistentwith societal norms (b) Action includes noaction some action or action to seek moreinformation

The first part of the definition refers to in-

HUMAN FACTORS

tellectual or discriminating abilities Thispart depends upon human capabilities tosense store retrieve and integrate informa-tion This function is what Van Dam (in Jen-sen and Benel 1977) calls the discriminatingability in professional pilots It is analogousto detectability Cd) in signal detection theoryIt is purely rational and could be statedmathematically If it were possible to sepa-rate this part of human judgment from thesecond part (which it is not) people wouldsolve problems in much the same way as acomputer does This is not to say that the de-cisions would be error free Probabilistic in-formation is used and the performance is de-pendent upon the amount type and accuracyof information stored as well as on inherent andlearned capabilities to process information

The second part of the definition refers tomotivational tendencies The emphasis inthis case is on the directional aspects of moti-vation rather than on intensity It says that apart of human judgment is based upon biasfactors (costs and payoffs) or tendencies touse less than rational information (defined bythe society) in choosing courses of action Thesociety would probably consider the use ofany information other than that required todefine the safety risk (eg conveniencemonetary gain gain in self-esteem adventureseeking etc) as less than rational This partof human judgment is analogous to the re-sponse criterion (m in signal detection theoryIt is what Van Dam (in Jensen and Benel1977) has called the response pattern of theprofessional pilot If properly developed thispart of human judgment would tend to haltthe use of information not directly related tothe safety of the flight and to direct the pilotsdecision toward the use of rational processes

JUDGMENT TRAINING

One of the most important questions to beaddressed after establishing the definition is

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--65

whether or not pilot judgment as definedcan be modified through training The pau-city of judgment-training guidelines in pilot-training and training-research literaturemight lead one to conclude that pilot judg-ment cannot be taught Literature and syllabicommonly used in flight-instructor coursescontain large sections on how to teach themotor skills of flying but offer very little onhow to teach pilot judgment (see Federal Avi-ation Administration 1977) The typicalprivate-pilot course offers a scattering ofjudgment-making instruction in the areas ofweather avoidance and power-plant emer-gencies but does not offer systematic judg-ment training

However there is evidence in aviation thatat least one form of judgment training ieassigning procedures for every conceivablesituation that might arise may be effectiveIn the military these are referred to asBoldface training procedures (ThorpeMartin Edwards and Eddowes 1976) Sim-ilar training procedures used by the airlineshave even been more successful than thoseused by the military (Gibson 1969 TransWorld Airlines 1969)

Looking outside the field of aviation onefinds other evidence indicating that judgmentmaking may be taught For example al-though signal detection theory (SDT) was notdesigned specifically to handle cognitivejudgments many of its methods can be usedto explain and perhaps even to modify pilotjudgment behavior SDT divides an indi-viduals decision behavior into two compo-nents representing his or her sensitivity (d)and response criterion or bias (f3roughlycorresponding to the two aspects of thejudgment definition previously stated

The sensitivity is affected both by the in-equality of the stimulus in the situation (sig-nal vs background noise) and the quality ofthe sensory apparatus of the observer In cog-nitive judgment this is the intellective com-

ponent It refers both to the strength of theevidence favoring the selection of one alterna-tive over another and to the knowledge andability of the pilot to use effective strategiesin gathering and processing the evidence

On the other hand the response criterionrepresen ts the level of excitation abovewhich the observer is willing to say signaIt is the amount of relevant information in thepresence of irrelevant information or noiseneeded to tip the decision one way or theother The criterion level (ie the amount ofrelevant information required) is influencedby motivation knowledge of the signalsprobability of occurrence and the costs andpayoffs attendant with a given response Incognitive judgment the response criterion isthe motivative component

The response criterion can be manipulatedthrough a wide range of values by adjustingprobabilities costs and payoffs (Birdsall1955) One can infer from the vast amount ofpsychophysical decision research that cogni-tive judgments can be modified in a similarway The flight instructor communicates de-cision biases attitudes risk tendencies con-sideration for passenger safety and pilotmotivation by example if not by design at alllevels of pilot training The method by whichthe instructor teaches these values is influ-enced by the instructors assignment of prob-abilities costs and payoffs to actions of thestudent

Although the SDT model assumes that thesensitivity component is quite stable for agiven individual there is a growing field ofresearch indicating that when dealing withthe intellective component of cognitivejudgment sensitivity can be modified as weiFor example attempts have been made todiscover the mental processes that are usedby expert judges such as stockbrokers live-stock judges and medical diagnosticians inmaking their decisions (Shanteau and Phelps1977 Slovic 1969 Anderson 1969 Hoffman

66--February 1982

Slovic and Rorer 1968) The thesis is that ifmodels of the mental processes used by theseexperts in decision making were availablethey could be used in training others to usesimilar processes In each of the areasstudied judgment training traditionally oc-curs over a fairly long apprenticeship pro-gram in which the trainee observes the ex-pert making decisions and learns by this ob-servation However as in aviation because ofthe complexity of the information used tomake decisions observation or even trial anderror are inefficient training methods

Research on the motivational aspect ofcognitive judgment also indicates that train-ing can have a beneficial effect The majorresearch efforts in this area are reported byJanis and Mann (1977) These authorsspeaking from a clinical perspective beginwith the assumption that psychological stressis a frequent cause of errors in decision mak-ing They say that stress arises from at leasttwo sources First the decision maker is con-cerned about the material and social lossesthat may be suffered from whichever courseof action is chosen including the costs offailing to live up to prior commitments Sec-ond the pilots reputation and self-esteem asa competent decision maker are at stake Themore severe the anticipated losses thegreater the stress

Janis and Mann have constructed aconflict-theory model of decision makingpostulating that the way difficult choices areresolved is determined by the presence orabsence of three conditions awareness ofrisks involved hope of finding a better so-lution and time available in which to makethe decision They have developed severalclinical procedures to improve decisionmaking under the titles awareness of ra-tionalizations emotional role playingbalance sheet and outcome psychodra-ma They report that these procedures havedemonstrated effectiveness in changing

HUMAN FACTORS

decision-making tendencies and in modifyingattitudes

Some Learning Principles

Because some well-established learningprinciples are misapplied in many trainingprograms a discussion of these principles asapplied to pilot judgment training is neededPerhaps the most popular principle is thatthe best way to learn an activity is to practicethat activity (Gagne 1962) The assumption isrooted in much of the educational literatureand is often identified by the catch-phraselearning by doing Gagne points out that itmay also be a generalization from research onthe conditioned response in which learningparticularly in animals appears to have oc-curred only after a response (practice) hasbeen made

However Gagne argues that practice is notan effective training method by itself even forthe acquisition of such motor skills as fieldgunnery He says that instruction about thecorrect sighting picture for ranging is moreeffective in bringing about improved perfor-mance than is practice on the task (Gagne1962 p 85) The point is that training shouldemphasize the principles and procedures(thought processes) involved and practiceshould be directed to take advantage of theseprinciples If this is the proper emphasis forteaching motor skills it is even more impor-tant in the teaching of judgment-makingskills which are more deeply rooted inthought processes

A second learning principle that is fre-quently misapplied in training situations isvariously called reinforcement feedback orknowledge of results during practice Thisprinciple has been found to be most effectivein choice behavior However Gagne pointsout that some manipulations that artificiallyimprove feedback during practice failed toshow reliably better transfer to the opera-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-67

tional environment and that others showednegative transfer perhaps because of alearned dependence on the artificial cuesApparently the form ofthe feedback is important

Any beginning flight student will say thatsuch usual feedback information as you didit right or you did it wrong is almost use-less The time interval between execution andfeedback may be long the feedback is oftencluttered with interfering information andthe trials themselves are often so complexthat the student may learn very little fromsuch a response by the flight instructor Astudent really needs to know why the re-sponse was right or wrong He or she needs toknow what rules should have been followedand if and how those rules were violated Al-though practice and rightwrong types offeedback may be useful in some training situ-ations they should be deemphasized in favorof these thought-oriented teaching principlesin all types of pilot training but especially injudgment training

A Systematic Approach to Training

The need for pilot judgment training hasbeen established for all levels of flight in-struction Without a systematic judgment-training program good pilot judgment is ac-quired by the cautions and the lucky overyears of flying experience in many variedsituations The primary task of aviationeducators using systematic judgment-train-ing techniques is to compress a lifetime offlying experience into a relatively short train-ing program to instill good pilot judgmentinto the emerging pilot

In addition a ground school might includeinstruction on information integration andsubjective probability estimation (Goldberg1968) In expert judges judgment-makingbehavior is characterized by chunking or theformation of clusters of stimulus attributesand response alternatives for economy in the

thought process Ground school studentscould be taught to use these processes in theirjudgment-making activity The instructorwould show how various types of probabilis-tic information such as weather forecastspredicted aircraft system malfunctions andpredicted air traffic control problems shouldbe combined when making flight decisions

The instructor could teach the student howto anticipate decisions that might have to bemade later as a result of immediate choicesof action Such anticipation permits thegathering of relevant information under lowerlevels of stress when errors are less frequentrather than later in the flight when theamount of time available to decide may be-come an error-causing factor This section ofground school could also include decision-making training using procedures suggestedby Janis and Mann (1977) such as balancesheet and emotional role playingComputer-assisted instruction An instruc-

tion technique that holds exceptional prom-ise for pilot judgment training and evalua-tion is computer-assisted instruction (CAl)The great advantage of CAl systems is thatthey can be used to teach principles and thento permit the student to participate indecision-making processes a highly effectivelearning technique (Fishbein and Ajzen1975) The disadvantage of these systems inthe past has been their limited availabilityand high cost However recent advances intechnology are making them available at arelatively low cost (Trollip and Ortony 1977)

Although CAl programs are available inseveral forms the dialogue systems that per-mit student-computer interaction unre-stricted by preset response alternatives (Al-pert and Bitzer 1970) show the greatestpotential for application to instruction injudgment making These systems dependupon a set of stored algorithms that are usedby the computer to construct a great varietyof responses to student questions In addition

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--65

whether or not pilot judgment as definedcan be modified through training The pau-city of judgment-training guidelines in pilot-training and training-research literaturemight lead one to conclude that pilot judg-ment cannot be taught Literature and syllabicommonly used in flight-instructor coursescontain large sections on how to teach themotor skills of flying but offer very little onhow to teach pilot judgment (see Federal Avi-ation Administration 1977) The typicalprivate-pilot course offers a scattering ofjudgment-making instruction in the areas ofweather avoidance and power-plant emer-gencies but does not offer systematic judg-ment training

However there is evidence in aviation thatat least one form of judgment training ieassigning procedures for every conceivablesituation that might arise may be effectiveIn the military these are referred to asBoldface training procedures (ThorpeMartin Edwards and Eddowes 1976) Sim-ilar training procedures used by the airlineshave even been more successful than thoseused by the military (Gibson 1969 TransWorld Airlines 1969)

Looking outside the field of aviation onefinds other evidence indicating that judgmentmaking may be taught For example al-though signal detection theory (SDT) was notdesigned specifically to handle cognitivejudgments many of its methods can be usedto explain and perhaps even to modify pilotjudgment behavior SDT divides an indi-viduals decision behavior into two compo-nents representing his or her sensitivity (d)and response criterion or bias (f3roughlycorresponding to the two aspects of thejudgment definition previously stated

The sensitivity is affected both by the in-equality of the stimulus in the situation (sig-nal vs background noise) and the quality ofthe sensory apparatus of the observer In cog-nitive judgment this is the intellective com-

ponent It refers both to the strength of theevidence favoring the selection of one alterna-tive over another and to the knowledge andability of the pilot to use effective strategiesin gathering and processing the evidence

On the other hand the response criterionrepresen ts the level of excitation abovewhich the observer is willing to say signaIt is the amount of relevant information in thepresence of irrelevant information or noiseneeded to tip the decision one way or theother The criterion level (ie the amount ofrelevant information required) is influencedby motivation knowledge of the signalsprobability of occurrence and the costs andpayoffs attendant with a given response Incognitive judgment the response criterion isthe motivative component

The response criterion can be manipulatedthrough a wide range of values by adjustingprobabilities costs and payoffs (Birdsall1955) One can infer from the vast amount ofpsychophysical decision research that cogni-tive judgments can be modified in a similarway The flight instructor communicates de-cision biases attitudes risk tendencies con-sideration for passenger safety and pilotmotivation by example if not by design at alllevels of pilot training The method by whichthe instructor teaches these values is influ-enced by the instructors assignment of prob-abilities costs and payoffs to actions of thestudent

Although the SDT model assumes that thesensitivity component is quite stable for agiven individual there is a growing field ofresearch indicating that when dealing withthe intellective component of cognitivejudgment sensitivity can be modified as weiFor example attempts have been made todiscover the mental processes that are usedby expert judges such as stockbrokers live-stock judges and medical diagnosticians inmaking their decisions (Shanteau and Phelps1977 Slovic 1969 Anderson 1969 Hoffman

66--February 1982

Slovic and Rorer 1968) The thesis is that ifmodels of the mental processes used by theseexperts in decision making were availablethey could be used in training others to usesimilar processes In each of the areasstudied judgment training traditionally oc-curs over a fairly long apprenticeship pro-gram in which the trainee observes the ex-pert making decisions and learns by this ob-servation However as in aviation because ofthe complexity of the information used tomake decisions observation or even trial anderror are inefficient training methods

Research on the motivational aspect ofcognitive judgment also indicates that train-ing can have a beneficial effect The majorresearch efforts in this area are reported byJanis and Mann (1977) These authorsspeaking from a clinical perspective beginwith the assumption that psychological stressis a frequent cause of errors in decision mak-ing They say that stress arises from at leasttwo sources First the decision maker is con-cerned about the material and social lossesthat may be suffered from whichever courseof action is chosen including the costs offailing to live up to prior commitments Sec-ond the pilots reputation and self-esteem asa competent decision maker are at stake Themore severe the anticipated losses thegreater the stress

Janis and Mann have constructed aconflict-theory model of decision makingpostulating that the way difficult choices areresolved is determined by the presence orabsence of three conditions awareness ofrisks involved hope of finding a better so-lution and time available in which to makethe decision They have developed severalclinical procedures to improve decisionmaking under the titles awareness of ra-tionalizations emotional role playingbalance sheet and outcome psychodra-ma They report that these procedures havedemonstrated effectiveness in changing

HUMAN FACTORS

decision-making tendencies and in modifyingattitudes

Some Learning Principles

Because some well-established learningprinciples are misapplied in many trainingprograms a discussion of these principles asapplied to pilot judgment training is neededPerhaps the most popular principle is thatthe best way to learn an activity is to practicethat activity (Gagne 1962) The assumption isrooted in much of the educational literatureand is often identified by the catch-phraselearning by doing Gagne points out that itmay also be a generalization from research onthe conditioned response in which learningparticularly in animals appears to have oc-curred only after a response (practice) hasbeen made

However Gagne argues that practice is notan effective training method by itself even forthe acquisition of such motor skills as fieldgunnery He says that instruction about thecorrect sighting picture for ranging is moreeffective in bringing about improved perfor-mance than is practice on the task (Gagne1962 p 85) The point is that training shouldemphasize the principles and procedures(thought processes) involved and practiceshould be directed to take advantage of theseprinciples If this is the proper emphasis forteaching motor skills it is even more impor-tant in the teaching of judgment-makingskills which are more deeply rooted inthought processes

A second learning principle that is fre-quently misapplied in training situations isvariously called reinforcement feedback orknowledge of results during practice Thisprinciple has been found to be most effectivein choice behavior However Gagne pointsout that some manipulations that artificiallyimprove feedback during practice failed toshow reliably better transfer to the opera-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-67

tional environment and that others showednegative transfer perhaps because of alearned dependence on the artificial cuesApparently the form ofthe feedback is important

Any beginning flight student will say thatsuch usual feedback information as you didit right or you did it wrong is almost use-less The time interval between execution andfeedback may be long the feedback is oftencluttered with interfering information andthe trials themselves are often so complexthat the student may learn very little fromsuch a response by the flight instructor Astudent really needs to know why the re-sponse was right or wrong He or she needs toknow what rules should have been followedand if and how those rules were violated Al-though practice and rightwrong types offeedback may be useful in some training situ-ations they should be deemphasized in favorof these thought-oriented teaching principlesin all types of pilot training but especially injudgment training

A Systematic Approach to Training

The need for pilot judgment training hasbeen established for all levels of flight in-struction Without a systematic judgment-training program good pilot judgment is ac-quired by the cautions and the lucky overyears of flying experience in many variedsituations The primary task of aviationeducators using systematic judgment-train-ing techniques is to compress a lifetime offlying experience into a relatively short train-ing program to instill good pilot judgmentinto the emerging pilot

In addition a ground school might includeinstruction on information integration andsubjective probability estimation (Goldberg1968) In expert judges judgment-makingbehavior is characterized by chunking or theformation of clusters of stimulus attributesand response alternatives for economy in the

thought process Ground school studentscould be taught to use these processes in theirjudgment-making activity The instructorwould show how various types of probabilis-tic information such as weather forecastspredicted aircraft system malfunctions andpredicted air traffic control problems shouldbe combined when making flight decisions

The instructor could teach the student howto anticipate decisions that might have to bemade later as a result of immediate choicesof action Such anticipation permits thegathering of relevant information under lowerlevels of stress when errors are less frequentrather than later in the flight when theamount of time available to decide may be-come an error-causing factor This section ofground school could also include decision-making training using procedures suggestedby Janis and Mann (1977) such as balancesheet and emotional role playingComputer-assisted instruction An instruc-

tion technique that holds exceptional prom-ise for pilot judgment training and evalua-tion is computer-assisted instruction (CAl)The great advantage of CAl systems is thatthey can be used to teach principles and thento permit the student to participate indecision-making processes a highly effectivelearning technique (Fishbein and Ajzen1975) The disadvantage of these systems inthe past has been their limited availabilityand high cost However recent advances intechnology are making them available at arelatively low cost (Trollip and Ortony 1977)

Although CAl programs are available inseveral forms the dialogue systems that per-mit student-computer interaction unre-stricted by preset response alternatives (Al-pert and Bitzer 1970) show the greatestpotential for application to instruction injudgment making These systems dependupon a set of stored algorithms that are usedby the computer to construct a great varietyof responses to student questions In addition

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

66--February 1982

Slovic and Rorer 1968) The thesis is that ifmodels of the mental processes used by theseexperts in decision making were availablethey could be used in training others to usesimilar processes In each of the areasstudied judgment training traditionally oc-curs over a fairly long apprenticeship pro-gram in which the trainee observes the ex-pert making decisions and learns by this ob-servation However as in aviation because ofthe complexity of the information used tomake decisions observation or even trial anderror are inefficient training methods

Research on the motivational aspect ofcognitive judgment also indicates that train-ing can have a beneficial effect The majorresearch efforts in this area are reported byJanis and Mann (1977) These authorsspeaking from a clinical perspective beginwith the assumption that psychological stressis a frequent cause of errors in decision mak-ing They say that stress arises from at leasttwo sources First the decision maker is con-cerned about the material and social lossesthat may be suffered from whichever courseof action is chosen including the costs offailing to live up to prior commitments Sec-ond the pilots reputation and self-esteem asa competent decision maker are at stake Themore severe the anticipated losses thegreater the stress

Janis and Mann have constructed aconflict-theory model of decision makingpostulating that the way difficult choices areresolved is determined by the presence orabsence of three conditions awareness ofrisks involved hope of finding a better so-lution and time available in which to makethe decision They have developed severalclinical procedures to improve decisionmaking under the titles awareness of ra-tionalizations emotional role playingbalance sheet and outcome psychodra-ma They report that these procedures havedemonstrated effectiveness in changing

HUMAN FACTORS

decision-making tendencies and in modifyingattitudes

Some Learning Principles

Because some well-established learningprinciples are misapplied in many trainingprograms a discussion of these principles asapplied to pilot judgment training is neededPerhaps the most popular principle is thatthe best way to learn an activity is to practicethat activity (Gagne 1962) The assumption isrooted in much of the educational literatureand is often identified by the catch-phraselearning by doing Gagne points out that itmay also be a generalization from research onthe conditioned response in which learningparticularly in animals appears to have oc-curred only after a response (practice) hasbeen made

However Gagne argues that practice is notan effective training method by itself even forthe acquisition of such motor skills as fieldgunnery He says that instruction about thecorrect sighting picture for ranging is moreeffective in bringing about improved perfor-mance than is practice on the task (Gagne1962 p 85) The point is that training shouldemphasize the principles and procedures(thought processes) involved and practiceshould be directed to take advantage of theseprinciples If this is the proper emphasis forteaching motor skills it is even more impor-tant in the teaching of judgment-makingskills which are more deeply rooted inthought processes

A second learning principle that is fre-quently misapplied in training situations isvariously called reinforcement feedback orknowledge of results during practice Thisprinciple has been found to be most effectivein choice behavior However Gagne pointsout that some manipulations that artificiallyimprove feedback during practice failed toshow reliably better transfer to the opera-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-67

tional environment and that others showednegative transfer perhaps because of alearned dependence on the artificial cuesApparently the form ofthe feedback is important

Any beginning flight student will say thatsuch usual feedback information as you didit right or you did it wrong is almost use-less The time interval between execution andfeedback may be long the feedback is oftencluttered with interfering information andthe trials themselves are often so complexthat the student may learn very little fromsuch a response by the flight instructor Astudent really needs to know why the re-sponse was right or wrong He or she needs toknow what rules should have been followedand if and how those rules were violated Al-though practice and rightwrong types offeedback may be useful in some training situ-ations they should be deemphasized in favorof these thought-oriented teaching principlesin all types of pilot training but especially injudgment training

A Systematic Approach to Training

The need for pilot judgment training hasbeen established for all levels of flight in-struction Without a systematic judgment-training program good pilot judgment is ac-quired by the cautions and the lucky overyears of flying experience in many variedsituations The primary task of aviationeducators using systematic judgment-train-ing techniques is to compress a lifetime offlying experience into a relatively short train-ing program to instill good pilot judgmentinto the emerging pilot

In addition a ground school might includeinstruction on information integration andsubjective probability estimation (Goldberg1968) In expert judges judgment-makingbehavior is characterized by chunking or theformation of clusters of stimulus attributesand response alternatives for economy in the

thought process Ground school studentscould be taught to use these processes in theirjudgment-making activity The instructorwould show how various types of probabilis-tic information such as weather forecastspredicted aircraft system malfunctions andpredicted air traffic control problems shouldbe combined when making flight decisions

The instructor could teach the student howto anticipate decisions that might have to bemade later as a result of immediate choicesof action Such anticipation permits thegathering of relevant information under lowerlevels of stress when errors are less frequentrather than later in the flight when theamount of time available to decide may be-come an error-causing factor This section ofground school could also include decision-making training using procedures suggestedby Janis and Mann (1977) such as balancesheet and emotional role playingComputer-assisted instruction An instruc-

tion technique that holds exceptional prom-ise for pilot judgment training and evalua-tion is computer-assisted instruction (CAl)The great advantage of CAl systems is thatthey can be used to teach principles and thento permit the student to participate indecision-making processes a highly effectivelearning technique (Fishbein and Ajzen1975) The disadvantage of these systems inthe past has been their limited availabilityand high cost However recent advances intechnology are making them available at arelatively low cost (Trollip and Ortony 1977)

Although CAl programs are available inseveral forms the dialogue systems that per-mit student-computer interaction unre-stricted by preset response alternatives (Al-pert and Bitzer 1970) show the greatestpotential for application to instruction injudgment making These systems dependupon a set of stored algorithms that are usedby the computer to construct a great varietyof responses to student questions In addition

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-67

tional environment and that others showednegative transfer perhaps because of alearned dependence on the artificial cuesApparently the form ofthe feedback is important

Any beginning flight student will say thatsuch usual feedback information as you didit right or you did it wrong is almost use-less The time interval between execution andfeedback may be long the feedback is oftencluttered with interfering information andthe trials themselves are often so complexthat the student may learn very little fromsuch a response by the flight instructor Astudent really needs to know why the re-sponse was right or wrong He or she needs toknow what rules should have been followedand if and how those rules were violated Al-though practice and rightwrong types offeedback may be useful in some training situ-ations they should be deemphasized in favorof these thought-oriented teaching principlesin all types of pilot training but especially injudgment training

A Systematic Approach to Training

The need for pilot judgment training hasbeen established for all levels of flight in-struction Without a systematic judgment-training program good pilot judgment is ac-quired by the cautions and the lucky overyears of flying experience in many variedsituations The primary task of aviationeducators using systematic judgment-train-ing techniques is to compress a lifetime offlying experience into a relatively short train-ing program to instill good pilot judgmentinto the emerging pilot

In addition a ground school might includeinstruction on information integration andsubjective probability estimation (Goldberg1968) In expert judges judgment-makingbehavior is characterized by chunking or theformation of clusters of stimulus attributesand response alternatives for economy in the

thought process Ground school studentscould be taught to use these processes in theirjudgment-making activity The instructorwould show how various types of probabilis-tic information such as weather forecastspredicted aircraft system malfunctions andpredicted air traffic control problems shouldbe combined when making flight decisions

The instructor could teach the student howto anticipate decisions that might have to bemade later as a result of immediate choicesof action Such anticipation permits thegathering of relevant information under lowerlevels of stress when errors are less frequentrather than later in the flight when theamount of time available to decide may be-come an error-causing factor This section ofground school could also include decision-making training using procedures suggestedby Janis and Mann (1977) such as balancesheet and emotional role playingComputer-assisted instruction An instruc-

tion technique that holds exceptional prom-ise for pilot judgment training and evalua-tion is computer-assisted instruction (CAl)The great advantage of CAl systems is thatthey can be used to teach principles and thento permit the student to participate indecision-making processes a highly effectivelearning technique (Fishbein and Ajzen1975) The disadvantage of these systems inthe past has been their limited availabilityand high cost However recent advances intechnology are making them available at arelatively low cost (Trollip and Ortony 1977)

Although CAl programs are available inseveral forms the dialogue systems that per-mit student-computer interaction unre-stricted by preset response alternatives (Al-pert and Bitzer 1970) show the greatestpotential for application to instruction injudgment making These systems dependupon a set of stored algorithms that are usedby the computer to construct a great varietyof responses to student questions In addition

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

68--February 1982

student responses are not limited to exactduplicates of prestored expected responsesThe program recognizes a variety of studentresponses and is able to proceed accordingly

Although practice and feedback principlesare frequently used concepts in CAl pro-grams they could be augmented by present-ing principles and reasons for taking certaincourses of action In judgment training thestudent could be presented with a flight situ-ation requiring judgment The response couldrequire listing all of the alternatives and thefactors affecting each Student pilots couldeven be asked to estimate the probability ofsuccess for each alternative

The computer after examining the studentsflight experience data (entered previously)and the stored accident statistics from simi-lar circumstances would respond with com-ments on the appropriateness of the studentsresponses the alternatives that may havebeen omitted and the principles that shouldhave been followed in making the decisionThe program could then branch to anotherproblem the difficulty of which would bebased on the level of judgment-making capa-bility evidenced by the students responses tothe previous problem

Complexity realism and time constraintsmight be included in the judgment-makingtask by the addition of a simple hand-controller and an airplane symbol with a mapon the screen The controllers purpose wouldbe to provide indications of progress toward adestination and the time available for the deci-sion not to provide flight control instruction

CAl has many advantages not commonlyassociated with other instructional systemsThe most important of these is individualiza-tion of instruction CAl can adapt to the spe-cific needs of the individual and interact athis or her current level of ability (Goldstein1974) Second the unencumbered reinforce-ment capabilities of CAl are a real benefit tothe student There are no ulterior motives to

HUMAN FACTORS

clash with those of the student Third CAlsystems do not require the presence of ateacher although it may be beneficial to haveone present for occasional consultationFourth they permit standardization of in-struction across a wide area One centralcomputer could potentially support termi-nals at every pilot instructional center in theUnited States at a relatively low cost No stu-dent would be handicapped by a poor in-structor who underscores weaknesses in thesimulation Fifth data gathered from studentresponses could be stored for as long as neces-sary or could be used in updating instruc-tional programs or evaluating individualpilot judgment-making capabilitiesFlight simulation In some ways judgment

training in a simulator environment would bemore cumbersome than in ground school orwith CAl because at least in current practiceit is up to the instructor to create the simu-lated flight situation primarily through ver-bal communication However if properlystructured the simulated flight environmentprovides an additional opportunity to teachthe principles of making judgments in asomewhat more realistic environment thanthat provided by ground school or CAL

Probably the best way to begin judgmenttraining in the simulator is to use the airlineapproach ie the teaching of procedures thatare to be followed in each situation that de-parts from normal flight This includes detect-ing system failure as well as establishingcourses of action to correct or counter systemfailures Principles as well as corrective pro-cedures would be taught according to thismethod and appropriate judgment perfor-mance measures could be developed

The simulator instruction could also in-clude the creation by the flight instructor ofjudgment-demanding situations that do notinvolve the failure of systems These situa-tions would demand decisions such aswhether to continue a flight into deteriorat-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--69

ing weather decisions about passenger de-mands for landing at an unfamiliar alternateairport decisions about weight and balanceconsidering field conditions density altitudeetc In all cases the instructor would ask thestudent to state several available alternativesand also to state which would be chosenThese situations could be developed fromNTSB accident briefs and they could be apart of the flight instructors simulatorjudgment-instruction packageThe airplane Of all the media available the

airplane is probably the most difficult to usefor direct systematic training in judgmentmaking because for the sake of safety con-venience and cost most judgment problemsmust be halted before the student sees thefinal consequences of a decision The studentfrequently must take the instructors wordthat a decision would have resulted in a safeor unsafe situation However the airplaneoffers special opportunites for instruction injudgment making because the environment ismore realistic it is more meaningful andtherefore it is more likely to cause a perma-nent behavioral change in the student thanare other training media

Everything that has been said about in-structor attitudes and approaches to judg-ment training is even more important whenactually flying the airplane Effective instruc-tion in the airplane requires a consistent dis-ciplined flight instructor who always followsthe rules that the student is expected to fol-low or provides a good explanation for de-viating from them It also requires that theinstructor follow the learning principlesstated earlier ie that practice and feedbackare beneficial only when accompanied by di-rection and explanation

Judgment-making instructions in theairplane could take the form of simulatedsituations created by the instructor that re-quire the use of judgment Such activitiescould be interspersed throughout the flight-

trammg program To some extent such in-struction is already being done throughtraining in simulated engine failures othersystem failures and stalls of all types Thistraining could be expanded to include manyof the hypothetical situations discussedabove Portions of such simulated situationscould be a part of every instructional flight

Through situational training techniquesflight instructors may be able to modify theresponse tendencies of their students byteaching them that it is not socially demean-ing to refuse to fly or to turn around in theface of deteriorating circumstances Suchsituations could be made to occur severaltimes during the students instruction pro-gram in the airplane There is a tendency inmany pilots to believe that they are responsi-ble to maintain a virile or courageous imagefor themselves and for their profession Pilotshave often said that it is most difficult to turnaround the first time Thus students need tobe taught how to avoid the tremendous socialpressure that a group of important passen-gers can exert In making all important deci-sions they need to know how to isolate them-selves from flight-naive passengers

Finally one of the pilots most difficultevaluations is the evaluation of ones ownskill knowledge and judgment-makingcapability relative to a proposed flight Al-though research is needed to determine waysto effectively evaluate ones own capabilitiessome guidelines can be suggested for its ap-plication First pilots should be taught to de-velop a list of personal limitations on flightprocedures based on an estimation of theirown capabilities Second pilots should betaught that these limitations are applicableto all flights regardless of passengers iden-tity or their willingness to pay for a morerisky choice These personal limitationsshould be invoked during a rational momentand the pilots resolution should be strongenough to withstand social pressures to de-

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

70-Fehruary 1982

viate from them either before or during theflight

JUDGMENT EVALUATION

Perhaps the most difficult part of any studyof human judgment is the evaluation of per-formance This is because much of what mustbe evaluated cannot be observed directly butmust be inferred by observing other relatedbehaviors From discussions with flight in-structors and pilot-examining personnel it isclear that judgment is not currently beingevaluated effectively (Jensen and Benel1977)

Although flight test guides published by theFAAspecify that civilian pilots are to be eval-uated for their judgment capabilities nodefinition of judgment is provided For thisevaluation examiners primarily dependupon the assessment of flight instructors whohave had the opportunity to examine theirstudents decision-making capability over agreater variety of circumstances However ininterviews with flight instructors Jensen andBenel (1977) found only one who admitted tohaving failed a student purely on the basis ofpoor judgment Although many said that theycould recognize poor judgment studentswere failed on the basis of a borderline per-formance of some other more clearly defined fly-ing maneuver involving skilled performance

Some ideas for judgment evaluation areoffered by Van Dam who directs a flightschool in which pilot judgment receives astrong emphasis (Jensen and Benel 1977) Inhis approach the evaluation begins withpsychological and intelligence testing prior toadmitting students for flight instruction Ini-tial impressions from these pretrainingexaminations are augmented with othersubjective indicators of judgment such asobvious effort and attention to instructionrelaxation division of attention re-sponse delays confidence capacity for

HUMAN FACTORS

problem solving and initiative In laterpilot training evidence of judgment devel-opment is seen through an eagerness tolearn or high motivation teachabilityadaptability and flexibility an intuitivequality in thinking or decision making apattern of good choices and application ofmargins and allowances

The requirements of pilot judgment evalua-tion are even broader than these Society ex-pects pilots to make decisions based on theinterests of passengers and property ownersTherefore judgment must also be evaluatedin an absolute sense against this poorly de-fined scale Thus there are three major di-mensions along which judgment should beevaluated each presents a unique problem tothe evaluator

(1) The assessment of judgment capabilities andtendencies prior to flight training

(2)The assessment of the effects of training onpilot judgment

(3)The assessment of the amount of trainingtransferred to the operational flying envi-ronment

Pretraining Evaluation

It is important from the standpoint of bothsafety and economics to identify prior toflight instruction those persons who mayhave difficulty with some aspects of flyingjudgment If such individuals could be iden-tified they could either be discouraged fromseeking flight training or their training pro-grams could be modified to offset this defi-ciency

Unfortunately on the basis of psychologi-cal testing research to date the predictedsuccess of such a pretraining evaluation pro-gram is not very good For examplepsychologists and others have made many at-tempts with little success to identify a gen-eral personality trait known as risk takingand to link this trait to accident proneness(Shealy 1974) Shealy found that if one wereto limit the scope of the test to specific situa-

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

PILOT JUDGMENT TRAINING AND EVALUATION February 1982--71

tions such as downhill skiing its predictivevalidity would increase greatly Thereforeefforts to develop pretraining pilot judgmentprediction tests should not be discouraged bythe limited success of the general tests In-stead efforts should be made to design anaviation-specific test of judgment with pre-dictive validity

Pretraining evaluations of judgment abilityin pilot-training candidates is a potentiallyuseful adjunct to the entire training andevaluation process Results from such testscould be used by trainers to adapt their pro-grams to emphasize training in areas iden-tified in these tests Flight instructors couldbe alerted to possible weaknesses in indi-vidual students and adapt their training ac-cordingly Tests that could identify risk-taking tendencies (Kogan and Wallach 1964Taylor and Dunnette 1974) tests that iden-tify accident proneness (Shaw and Sichel1971) and situation-specific tests are poten-tially useful in this application

Training Evaluation

The second major dimension along whichpilot judgment must be evaluated is an as-sessment of the amount of change in thepilots judgment performance that resultsfrom training This measure would providean indication of the value of the training pro-gram as well as an indication of individualstudent progress

The development of clearly definedjudgment-evaluation criteria presents thegreatest challenge to effective evaluation ofpilot judgment in all phases of pilot trainingTo ensure that evaluations are made alongthe same dimensions as those along which thetraining is conducted the development ofthese criteria should be based on preestab-lished behavioral objectives Judgmentcriteria should consist of positive statementsof acceptable pilot judgment-making be-havior for each major area of flight activity

Similar criteria could be developed for everymajor maneuver taught These could begraded by the instructor together with evalu-ations of knowledge and skill each time themaneuvers are attempted

In pilot training for each level of pilot ex-perience certain judgment proficiency levelscould be objectively specified The instructoror examiner who evaluates the judgmentswould have a range of acceptable perfor-mances also objectively specified Evalua-tion of pilot judgment would be a matter ofcomparing performance against the estab-lished criteria in carefully structured situations

The critical point for evaluating judgmentin a national system is the use of the samecriteria by all judges as well as by the pilotsthemselves One way to ensure standardiza-tion of judgment evaluations is to use anationwide CAl system to administer tests atspecific times during each students trainingprogram Results of such tests could be usedto modify the individual students training orthe training program as a whole

Transfer Evaluation

The final dimension along which pilotjudgment must be evaluated is an assessmentof the amount of training that is transferredto the operational flight environment Thismeans that students who have received spe-cial judgment training would be comparedwith those who have not received such train-ing after both groups have moved into theoperational flight environment The results ofthis evaluation are used to modify both stu-dent selection criteria (or pretraining exami-nations) and program need assessments

The criteria for such an evaluation are ba-sically the same as those used in trainingevaluations except that they would be morehighly influenced by societal demands Indi-ces of the transfer of judgment could includethe number of accidents or incidents due to

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

72--February 1982

faulty pilot judgment compared across thetwo groups

Operationalizing Judgment Evaluations

The definition of pilot judgment has twocomponents discrimination among situa-tional dimensions and response selectionBoth components must be evaluated Toop(rationalize these components for use inany specific training or testing situation theevaluator might ask the following questions

(I) For discriminative judgment Did the studentconsider all of the available alternatives Wasall of the relevant information considered andassigned proper weights Was the relevantinformation integrated efficiently before thechoice was made

(2) For response selection tendencies Did thestudent exhibit any tendency to consider fac-tors other than safety (such as self-esteemadventure or social pressure) in making theresponse selection Was the student highlyprone to use semirelevant factors such as fi-nancial gain or convenience in situationswhere safety should have been the primaryconsideration

The initial step toward an operationalevaluation of pilot judgment for experiencedpilots was taken in recent study by Flathers(1980) In this study conjoint measurementtechniques were used to establish the worthfunctions for four factors affecting a diversiondecision during a paper and pencil simula-tion of an alternator failure in instrumentflight conditions The four factors were airtraffic control service (radar vs no radar) theweather at the airport (ceiling of 1000 or 500feet) the time to fly to the airport (15 vs 30min) and the best approach facilities (ILS vsADF) at the alternate airport

Flathers reports that these techniques ef-fectively discriminate between AIP and in-strument rated pilots in terms of the valueplaced on certain of these four factors Simi-lar discriminations were found as a functionof training backgrounds type of flying mostcommonly done and level of pilot ability todiagnose flight problems as determined froma separate test Each of these are primarily

HUMAN FACTORS

measures of the discriminative component ofjudgment

Flathers also measured the response selec-tion tendencies of the pilot In this test pilotswere asked to state how many of their previ-ously selected top airport choices would beavoided in favor of a more risky choice to takeadvantage of field maintenance facilities Al-though in this case the results were not statis-tically significant they do indicate that pri-vate and commercial pilots were more likelyto take a greater risk than were ATP pilots

The Flathers study shows that objectivetests can be developed to measure both as-pects of pilot judgment In all likelihoodgreater fidelity in the testing procedure couldbe found using computers or flight sim-ulators Although the first steps are beingtaken a great deal of research still needs to bedone to refine these tests and to validate theirresults in operational environments

The proper evaluation of pilot judgmentrequires more of the evaluator than just anoccasional passing glance at the instrumentpanel It requires the careful structuring ofthe situation requiring the exercise of judg-ment and a careful examination of the stu-dents actions Each evaluation should beconsidered a training occasion as well and assuch feedback should be given to the studentconcerning all aspects of the decisional situa-tion Evaluations of this sort place high de-mands on the flight instructor neverthelessthey seem to be warranted in view of the highnumber of fatalities caused by faulty judgment

ACKNOWLEDGMENTS

This researchwas sponsoredby the SystemsResearchand DevelopmentServiceoftheFederalAviationAdminis-trationPatrickRussellwas theContractingOfficersRep-resentativeThe author wishesto thank RussBenelJimFinneganMikeKelleyand RobDurstfor their assistancein this program

REFERENCES

AlpertDand Bitzer01 Advancesin computer-basededucationScience 1970 167 1582-1590

AndersonN H Commenton analysis-of-variancemodel

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149

PILOT JUDGMENT TRAINING AND EVALUATION February 1982-73

for assignment of configural cue utilization in clinicaljudgment Psychological Bulletin 19697263-65

Birdsall T G The theory of signal detectability In HQuastler (Ed) Information theory in psychology Prob-lems and methods Glencoe IL Free Press 1955391-402

Federal Aviation Administration Aviation instructorshandbook Washington DC US Government PrintingOffice No AC 61-411977

Fishbein M and Ajzen I Belief attitude intention andbehavior An introduction to theory and research Read-ing MA Addison-Wesley 1975

Flathers G W A study of decision-making behavior of air-craft pilots deviating from a planned flight Unpub-lished masters thesis Ohio State University Depart-ment of Industrial and Systems Engineering 1980

Gagne R M Military training and principles of learningAmerican Psychologist 1962 [7 83-91

Gibson J H Optimized flight crew training A step to-ward safer operations Fort Worth TX American Air-lines Flight Training Academy 1969

Goldberg L R Simple models or simple processes Someresearch on clinical judgments American Psychologist196823 483-496

Goldstein I L Training Program development and evalua-tion Belmont CA Wadsworth 1974

Hoffman P J Slovic P and Rorer L G An analysis ofvariance model for the assessment of configural cueutilization in clinical judgment Psychological Bulletin196869338-349

Janis I and Mann L Decision making A psychologicalanalysis of conflict choice and commitment New YorkFree Press 1977

Jensen R S and Bene R A Judgment evaluation andinstruction in civil pilot training Springfield VA Na-tional Technical Information Service Final ReportFAA-RD-78-24 1977

Kogan N and Wallach M Risk taking A study of cogni-tion and personality New York Holt Rinehart andWinston 1964

Roscoe S N Aviation Psychology Ames IA Iowa StateUniversity Press 1980 p 175

Shanteau J and Phelps R H Judgment and swine Ap-proaches and issues in applied judgment analysis InM F Kaplan and S Schwartz (Eds) Human iudgmentand decision processes Applications in problem settingsNew York Academic Press 1977

Shaw L and Sichel H SAccident proneness Research inthe occurrence causation and prevention of road acci-dents New York Pergamon 1971

Shealey J E Risk-taking in skilled task performance Pro-ceedings of the Eighteenth Annual Meeting of the HumanFactors Society Santa Monica CA Human Factors So-ciety 1974

Siovic P Analyzing the expert judge A descriptive studyof stockbrokers decision processes Journal of AppliedPsychology 196953 255-263

Taylor R N and Dunnette M D Influence of dog-matism risk-taking propensity and intelligence ondecision-making strategies for a sample of industrialmanagers Journal of Applied Psychology 1974 59420-423

Thorpe J A Martin E L Edwards B J and EddowesE E Situational emergency training F-15 emergencyprocedures training program Williams Air Force BaseAZ Technical Report No AFHRL-TR-76-47(1) June1976

Trans World Airlines Flight simulator evaluation KansasCity MO Author Flight Operations Training Depart-ment 1969

Trollip S R and Ortony A Real-time simulation incomputer-assisted instruction Instruction Science19776 135-149