William Rantz Western Michigan University Department of Aviation Sciences March 16, 2009

COMPARING THE ACCURACY OF COMPARING THE ACCURACY OF PERFORMING DIGITAL AND PAPER PERFORMING DIGITAL AND PAPER CHECKLISTS USING A FEEDBACK CHECKLISTS USING A FEEDBACK INTERVENTION PACKAGE DURING INTERVENTION PACKAGE DURING NORMAL WORKLOAD CONDITIONS NORMAL WORKLOAD CONDITIONS

IN SIMULATED FLIGHTIN SIMULATED FLIGHT

William RantzWilliam RantzWestern Michigan UniversityWestern Michigan UniversityDepartment of Aviation SciencesDepartment of Aviation Sciences

March 16, 2009 March 16, 2009

22

OverviewOverview Rationale & PurposeRationale & Purpose Location & DurationLocation & Duration Frasca Flight Training DeviceFrasca Flight Training Device ParticipantsParticipants Paper-Digital Checklists & Flight PatternPaper-Digital Checklists & Flight Pattern Dependent Variables Dependent Variables Independent Variables & Integrity of IVsIndependent Variables & Integrity of IVs Experimental DesignExperimental Design ResultsResults Inferential ResultsInferential Results Discussion & Future ResearchDiscussion & Future Research

33

RationaleRationale

Most common error cited in LOSA dataMost common error cited in LOSA data Observational dataObservational data 54% of errors54% of errors (Helmreich et al., 2001)(Helmreich et al., 2001)

Contributing factor to numerous Contributing factor to numerous accidentsaccidents Improper configuration of aircraftImproper configuration of aircraft (NTSB, 1969, (NTSB, 1969,

1975, 1982, 1988a, 1988b, 1989, 1990, 1997, 1998, 2001, 2002, 2003a, 1975, 1982, 1988a, 1988b, 1989, 1990, 1997, 1998, 2001, 2002, 2003a, 2003b, 2004a, 2004b, 2006, 2007a, 2007b, 2008a, 2008b, 2008c, 2003b, 2004a, 2004b, 2006, 2007a, 2007b, 2008a, 2008b, 2008c, 2008d)2008d)

Improper use of checklist Improper use of checklist (Adamski & Stahl, 1997 (Adamski & Stahl, 1997 Degani, 1992, 2002; Diez, Boehm-Davis, & Holt, 2003; Federal Aviation Degani, 1992, 2002; Diez, Boehm-Davis, & Holt, 2003; Federal Aviation Administration [FAA], 1995, 2000; Lautmann and Gallimore, 1987; Administration [FAA], 1995, 2000; Lautmann and Gallimore, 1987; Turner, 2001)Turner, 2001)

44

RationaleRationale Digital Checklist ImportanceDigital Checklist Importance

Assertion: Errors prone to paper are reduced by digital Assertion: Errors prone to paper are reduced by digital formatformat

• Items skippedItems skipped• Losing place when distractedLosing place when distracted• Incorrect switch selectedIncorrect switch selected• Item incorrectly confirmed completeItem incorrectly confirmed complete• Excessive psychomotor workload fumbling with paperExcessive psychomotor workload fumbling with paper• Unreadable text due to low illuminationUnreadable text due to low illumination• Subsequent checklist accomplished before critical flight phaseSubsequent checklist accomplished before critical flight phase• All checklists omittedAll checklists omitted

Enhanced flow of branched sequences-abnormal checklistsEnhanced flow of branched sequences-abnormal checklists These errors not experimentally confirmedThese errors not experimentally confirmed (Boorman, 2001)(Boorman, 2001)

Improved checklist performance using graphic Improved checklist performance using graphic feedbackfeedback

Normal work load in simulated environmentNormal work load in simulated environment Paper checklistsPaper checklists (Rantz, Dickinson, Sinclair, Van Houten, 2009)(Rantz, Dickinson, Sinclair, Van Houten, 2009)

55

PurposePurpose

To examine whether pilots would complete To examine whether pilots would complete airplane digital checklists more accurately airplane digital checklists more accurately when they received post-flight graphic and when they received post-flight graphic and verbal feedbackverbal feedback

No study compared the accuracy of No study compared the accuracy of traditional paper against emergent digital traditional paper against emergent digital checklistschecklists

Only second study in aviation to attempt to Only second study in aviation to attempt to increase checklist accuracy using increase checklist accuracy using experimental manipulation of IVsexperimental manipulation of IVs

66

Location & DurationLocation & Duration

The simulation laboratory is located in The simulation laboratory is located in a hanger at WMU’s Aviation Education a hanger at WMU’s Aviation Education Center in Battle Creek, MICenter in Battle Creek, MI

Data collection took approximately 64 Data collection took approximately 64 sessionssessions July 29, 2008 through March 11, 2009July 29, 2008 through March 11, 2009 256 flight trials256 flight trials

77

Frasca 241 Flight Training Frasca 241 Flight Training Device –Cirrus SR20Device –Cirrus SR20

88

Frasca 241 Instructor Station Frasca 241 Instructor Station Cirrus SR20Cirrus SR20

99

Cirrus SR20 Primary Flight Cirrus SR20 Primary Flight Display (PFD)Display (PFD)

1010

Cirrus SR20 Multi Function Cirrus SR20 Multi Function Display (MFD)Display (MFD)

1111

Cirrus SR20 Multi Function Cirrus SR20 Multi Function Display (MFD) ChecklistDisplay (MFD) Checklist

1212

Observation AreaObservation Area

1313

ParticipantsParticipants

6 WMU flight students6 WMU flight students Males Males Private Pilot CertificatePrivate Pilot Certificate 125 minimum flight hours125 minimum flight hours Instrument ratedInstrument rated Average 186 total time flight hoursAverage 186 total time flight hours Average 80 hours total time in Cirrus Average 80 hours total time in Cirrus

Aircraft or FTDAircraft or FTD

1414

Digital Flight ChecklistDigital Flight Checklist

1515

Paper Flight ChecklistPaper Flight Checklist

1616

Null Hypotheses Null Hypotheses

1) There is no intervention effect with 1) There is no intervention effect with either paper or digital checklists.either paper or digital checklists. Tested at both the individual and group Tested at both the individual and group

levellevel 2) There is no difference between 2) There is no difference between

paper or digital checklist accuracy paper or digital checklist accuracy during all phases. during all phases.

1717

Main Dependent Variable Main Dependent Variable

The number of The number of paperpaper checklist items checklist items completed correctly per flightcompleted correctly per flight

The number of The number of digitaldigital checklist items checklist items completed correctly per flight completed correctly per flight

1818

Secondary Dependent Variable 1Secondary Dependent Variable 1

The percentage of total errors for each of the The percentage of total errors for each of the eight flight segments using the eight flight segments using the paperpaper checklist during each experimental phase checklist during each experimental phase (baseline, feedback, and reversal) per (baseline, feedback, and reversal) per participantparticipant

The percentage of total errors for each of the The percentage of total errors for each of the eight flight segments using the eight flight segments using the digitaldigital checklist during each experimental phase checklist during each experimental phase (baseline, feedback, and reversal) per (baseline, feedback, and reversal) per participant participant

1919


The percentage of baseline trials The percentage of baseline trials participants performed each of the participants performed each of the paperpaper checklist items incorrectly checklist items incorrectly

The percentage of baseline trials The percentage of baseline trials participants performed each of the participants performed each of the digitaldigital checklist items incorrectly checklist items incorrectly

2020


The percentage of baseline trials The percentage of baseline trials participants omitted participants omitted paperpaper checklist checklist items items

The percentage of baseline trials The percentage of baseline trials participants omitted participants omitted digitaldigital checklist checklist items items

2121


Percentage of timing errors Percentage of timing errors participants performed participants performed paperpaper checklist checklist segments too early or too latesegments too early or too late

Percentage of timing errors Percentage of timing errors participants performed participants performed digitaldigital checklist checklist segments too early or too latesegments too early or too late

2222

Experimental PhasesExperimental Phases

Baseline Baseline Only technical feedback of flight performance was givenOnly technical feedback of flight performance was given

Checklist Graphic Feedback & Vocal Praise Checklist Graphic Feedback & Vocal Praise Technical feedback of flight performance Technical feedback of flight performance Graphic feedback on the total number of checklist items Graphic feedback on the total number of checklist items

completed correctly per flight completed correctly per flight Graphic feedback on the number of items completed Graphic feedback on the number of items completed

correctly, completed incorrectly, and omitted for each of the correctly, completed incorrectly, and omitted for each of the eight flight segments per flight eight flight segments per flight

Vocal praise for any improvementVocal praise for any improvement

ReversalReversal Only technical feedback of flight performance was given Only technical feedback of flight performance was given

60-90 Day Data Probe60-90 Day Data Probe No Feedback was givenNo Feedback was given

23232323

Technical FeedbackTechnical Feedback

2424

Graphic Feedback Received by Participant:Graphic Feedback Received by Participant:Total Checklist Items CorrectTotal Checklist Items Correct

Participant 03 - Items Completed CorrectlyClosed Data Points Represent Digital ChecklistOpen Data Points Represent Paper Checklist

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Trials

Nu

mb

er o

f C

hec

klsi

t It

ems

2525

Graphic Feedback Received by Participant:Graphic Feedback Received by Participant:Flight Segments Flight Segments (Total-Correct-Incorrect-Omitted)(Total-Correct-Incorrect-Omitted)

P6S7T1DChecklist Items Per Segment

0

10

20

30

40

50

60

70

Before Takeoff Normal Takeoff Climb Cruise Descent Before Landing After Landing Shut Down Total

Total Items

Correct

Incorrect

Omitted

2626

Inter Observer AgreementInter Observer Agreement

Inter-observer agreement for Inter-observer agreement for correct correct and incorrect itemsand incorrect items was an average of was an average of 95% with a range of 79% to 100%. 95% with a range of 79% to 100%.

Inter-observer agreement for Inter-observer agreement for omitted omitted itemsitems was an average of 97% with a was an average of 97% with a range of 63% to 100%.range of 63% to 100%.

2727

Integrity of the IVIntegrity of the IV

Technical flight and checklist feedback Technical flight and checklist feedback were read from prepared scripts were read from prepared scripts

Participants were asked to initial the Participants were asked to initial the technical flight diagrams and the technical flight diagrams and the checklist feedback graphs and returned checklist feedback graphs and returned to the experimenter. to the experimenter.

Integrity of IV = 100%Integrity of IV = 100%

2828

Experimental DesignExperimental Design

A multiple baseline with reversal A multiple baseline with reversal design across paired individualsdesign across paired individuals Initial phase change occurred when Initial phase change occurred when

performance was judged as stable upon performance was judged as stable upon visual inspection visual inspection

Reversal phase change occurred within 3 Reversal phase change occurred within 3 consecutive (paper or digital) trials consecutive (paper or digital) trials exceeded 95% items correctexceeded 95% items correct

Probe phase occurred between 60-90 days Probe phase occurred between 60-90 days past last reversal trialpast last reversal trial

2929

Results Results

Individually and grouped-all participants Individually and grouped-all participants using both paper and digital checklists using both paper and digital checklists increased performance accuracy over increased performance accuracy over baseline when post-flight checklist feedback baseline when post-flight checklist feedback and praise was addedand praise was added

Improvements in performance remained near Improvements in performance remained near perfect during intervention withdrawalperfect during intervention withdrawal

Improvements declined slightly within the 60-Improvements declined slightly within the 60-90 day probe period90 day probe period

Results were statistically significantResults were statistically significant

3030

Results-Figure 5Results-Figure 5 continuedcontinued

Average percentage of paper checklist Average percentage of paper checklist items completed correctly increased items completed correctly increased from 38% during the baseline phase to from 38% during the baseline phase to 90% during the intervention phase 90% during the intervention phase

Average percentage of digital checklist Average percentage of digital checklist items completed correctly increased items completed correctly increased from 39% during the baseline phase to from 39% during the baseline phase to 89% during the intervention phase89% during the intervention phase

3131

Results-Figure 5Results-Figure 5 continuedcontinued

The average percentage of paper The average percentage of paper checklist items completed correctly checklist items completed correctly was nearly 100% during the return to was nearly 100% during the return to baseline conditionbaseline condition

The average percentage of digital The average percentage of digital checklist items completed correctly checklist items completed correctly was 99% during the return to baseline was 99% during the return to baseline conditioncondition

3232

Results-Figure 5Results-Figure 5continuedcontinued

The average percentage of paper checklist The average percentage of paper checklist items completed correctly was 97% during items completed correctly was 97% during the probe conditionthe probe condition

The average percentage of digital checklist The average percentage of digital checklist items completed correctly was 96% during items completed correctly was 96% during the probe conditionthe probe condition

3% decrement in paper performance after 2-3 3% decrement in paper performance after 2-3 monthsmonths

4% decrement in digital performance after 2-4% decrement in digital performance after 2-3 months3 months

33333333

ResultsResults


0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Trials

Nu

mb

er o

f C

hec

klis

t It

ems

34343434

ResultsResults


0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Trials

Nu

mb

er o

f C

hec

klsi

t It

ems

35353535

ResultsResults


0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Trials

Nu

mb

er o

f C

hec

klsi

t It

ems

36363636

ResultsResults


0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Trials

Nu

mb

er o

f C

hec

klis

t It

ems

37373737

ResultsResults


0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Trials

Nu

mb

er o

f C

hec

klis

t It

ems

38383838

ResultsResults


0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Trials

Nu

mb

er o

f C

hec

klis

t It

ems

3939

Results-Paper Checklist Results-Paper Checklist SegmentsSegments-Figure 6-Figure 6

During all flights using paper checklists, 2,451 During all flights using paper checklists, 2,451 total errors were observedtotal errors were observed

The average percentage of paper checklist The average percentage of paper checklist segment errors was highest for the normal take-segment errors was highest for the normal take-off segment (75%, range = 56.67% - 84.44%)off segment (75%, range = 56.67% - 84.44%)

The average percentage of paper checklist The average percentage of paper checklist segment errors was lowest for the after landing segment errors was lowest for the after landing segment (50%, range = 0% -100%)segment (50%, range = 0% -100%)

Please see Figure 6 hand out Please see Figure 6 hand out

4040

ResultsResults-Digital Checklist -Digital Checklist Segments-Figure 7Segments-Figure 7

During all flights using digital checklists, 2,562 During all flights using digital checklists, 2,562 total errors were observedtotal errors were observed

The average percentage of digital checklist The average percentage of digital checklist segment errors was highest for the normal take-segment errors was highest for the normal take-off segment (71%, range = 56.67% - 88.89%)off segment (71%, range = 56.67% - 88.89%)

The average percentage of digital checklist The average percentage of digital checklist segment errors was lowest for the after landing segment errors was lowest for the after landing segment (50%, range = 1.85% -100%)segment (50%, range = 1.85% -100%)

Please see Figure 7 hand out Please see Figure 7 hand out

4141

Results-Figure 6 & 7 SummationResults-Figure 6 & 7 Summation

Generally, the percentage of errors by Generally, the percentage of errors by flight segment for both paper and digital flight segment for both paper and digital checklists varied across participants and checklists varied across participants and flight segmentsflight segments

Errors decreased considerably for all Errors decreased considerably for all participants during intervention. participants during intervention.

Errors were very low during reversalErrors were very low during reversal Errors increased slightly after 60 daysErrors increased slightly after 60 days

4242

Results-Paper Checklist Total Results-Paper Checklist Total Incorrect Items-Table 1Incorrect Items-Table 1

Percentages of incorrect items that are 50% or greater are Percentages of incorrect items that are 50% or greater are shaded for each participant. Also, the checklist item name is shaded for each participant. Also, the checklist item name is shaded if the percentage of error was 50% or greater for four or shaded if the percentage of error was 50% or greater for four or more participants more participants

The highest frequency of errors, across nearly all participants The highest frequency of errors, across nearly all participants occurred for nine items in the before take-off segment: occurred for nine items in the before take-off segment: ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PIOTOT HEAT, NAV LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PIOTOT HEAT, NAV LIGHT, AND LANDING LIGHT. (AVG 86% n=9)AND LANDING LIGHT. (AVG 86% n=9)

Four items in the normal takeoff segment were problematic for Four items in the normal takeoff segment were problematic for nearly all participants: POWER LEVER, ENGINE PARAMETERS, nearly all participants: POWER LEVER, ENGINE PARAMETERS, BRAKES, ELEVATOR CONTROL. (AVG 92% n=4) BRAKES, ELEVATOR CONTROL. (AVG 92% n=4)

Please see Table 1 hand outPlease see Table 1 hand out

4343

Results-Digital Checklist Total Results-Digital Checklist Total Incorrect Items-Table 2Incorrect Items-Table 2

Percentages that are 50% or greater are shaded for each Percentages that are 50% or greater are shaded for each participant. Also, the checklist item name is shaded if the participant. Also, the checklist item name is shaded if the percentage of error was 50% or greater for four or more percentage of error was 50% or greater for four or more participants participants

The highest frequency of errors, across nearly all participants The highest frequency of errors, across nearly all participants occurred for the same nine items in the before take-off occurred for the same nine items in the before take-off segment: ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING segment: ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PITOT HEAT, NAV LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PITOT HEAT, NAV LIGHT, AND LANDING LIGHT. (AVG 91% n=9)LIGHT, AND LANDING LIGHT. (AVG 91% n=9)

Four items in the normal takeoff segment were problematic for Four items in the normal takeoff segment were problematic for nearly all participants: POWER LEVER, ENGINE PARAMETERS, nearly all participants: POWER LEVER, ENGINE PARAMETERS, BRAKES, ELEVATOR CONTROL. (AVG 84% n=4) BRAKES, ELEVATOR CONTROL. (AVG 84% n=4)


4444

Results-Paper Checklist Total Results-Paper Checklist Total Omitted Items-Table 3Omitted Items-Table 3

Percentages of omitted items that are 50% or greater Percentages of omitted items that are 50% or greater are shaded for each participant. Also, the checklist are shaded for each participant. Also, the checklist item name is shaded if the percentage of omission item name is shaded if the percentage of omission was 50% or greater for four or more participants was 50% or greater for four or more participants

There were no shaded omission errors for any There were no shaded omission errors for any checklist item either paper or digitalchecklist item either paper or digital

There were no shaded omission errors for any There were no shaded omission errors for any participant either paper or digitalparticipant either paper or digital

No omitted item resulted in a crash or incidentNo omitted item resulted in a crash or incident There were only a random selection of omission There were only a random selection of omission

items across many segments that P1 (AVG 19%), P2 items across many segments that P1 (AVG 19%), P2 (AVG 13%), and P4 (AVG 18%) did not perform. (AVG 13%), and P4 (AVG 18%) did not perform.


4545

Results-Digital Checklist Total Results-Digital Checklist Total Omitted Items-Table 4Omitted Items-Table 4

Percentages of omitted digital checklist items that are 50% or greater Percentages of omitted digital checklist items that are 50% or greater are shaded for each participant. Also, the digital checklist item name are shaded for each participant. Also, the digital checklist item name is shaded if the percentage of omission was 50% or greater for four or is shaded if the percentage of omission was 50% or greater for four or more participants more participants

There were no shaded omission errors for any checklist item either There were no shaded omission errors for any checklist item either paper or digitalpaper or digital

There were no shaded omission errors for any participant either paper There were no shaded omission errors for any participant either paper or digitalor digital

No omitted item resulted in a crash or incidentNo omitted item resulted in a crash or incident P1 omitted a high percentage (AVG 24%) of items during the CRUISE, P1 omitted a high percentage (AVG 24%) of items during the CRUISE,

DESCENT, and AFTER LANDING segmentsDESCENT, and AFTER LANDING segments P4 omitted a high percentage (24%) randomly across several P4 omitted a high percentage (24%) randomly across several

segmentssegments P2 omitted a high percentage (15%) of the same items cited as done P2 omitted a high percentage (15%) of the same items cited as done

incorrectly in both the paper and digital during the before takeoff incorrectly in both the paper and digital during the before takeoff segment (ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, segment (ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PITOT HEAT, NAV LIGHT, AND ANNUNCIATOR LIGHT, VOLTAGE, PITOT HEAT, NAV LIGHT, AND LANDING LIGHT)LANDING LIGHT)


4646

Results-Paper Checklist Segment Timing Results-Paper Checklist Segment Timing Errors in Baseline-Figure 8Errors in Baseline-Figure 8

Descent (43%)Descent (43%) Before Landing (39%)Before Landing (39%) Climb (38%)Climb (38%) Please see Figure 8 hand outPlease see Figure 8 hand out

4747

Results-Digital Checklist Segment Timing Results-Digital Checklist Segment Timing Errors in Baseline-Figure 9Errors in Baseline-Figure 9

Before Landing (42%)Before Landing (42%) Descent (39%)Descent (39%) Climb (22%)Climb (22%) Please see Figure 9 hand outPlease see Figure 9 hand out

4848

Individual Inferential Statistical AnalysisIndividual Inferential Statistical Analysis

General time-series intervention General time-series intervention regression modelingregression modeling Huitema and McKean (1998, 2000a,b) and Huitema and McKean (1998, 2000a,b) and

McKnight, McKean, and Huitema (2000)McKnight, McKean, and Huitema (2000) Bootstrap based time-series regression Bootstrap based time-series regression

methodmethod estimate estimate parameters of parameters of individual’s behavior individual’s behavior

AVG Baseline, Beta2, Beta3, AVG AVG Baseline, Beta2, Beta3, AVG remaining Intervention, AVG Reversal, remaining Intervention, AVG Reversal, AVG ProbeAVG Probe

49494949

Beta SourceBeta Source


0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Trials

Nu

mb

er o

f C

hec

klis

t It

ems

5050

Group Inferential Statistical AnalysisGroup Inferential Statistical Analysis

The purpose of this analysis was to provide The purpose of this analysis was to provide an overall evaluation of the effects of the an overall evaluation of the effects of the interventions for the group of six pilots over interventions for the group of six pilots over timetime

Previous time-series regression estimates Previous time-series regression estimates parameters of individual’s behavior are used parameters of individual’s behavior are used as DVsas DVs

One sample T-test used to evaluate One sample T-test used to evaluate hypothesis (no difference in intervention or hypothesis (no difference in intervention or phase changes)phase changes)

Ho: AVG mean of BetaHo: AVG mean of Beta1 1 = 0= 0

5151

Checklist Inferential Statistical AnalysisChecklist Inferential Statistical Analysis

Difference between Paper & Digital raw Difference between Paper & Digital raw scoresscores

Double bootstrap based time-series Double bootstrap based time-series regression methodregression method t to evaluate the o evaluate the hypothesis of zero difference between digital hypothesis of zero difference between digital and paper feedbackand paper feedback Based on an analysis not reported here, does not Based on an analysis not reported here, does not

differ as a function of subject or phase.differ as a function of subject or phase. Non significant resultsNon significant results

Avg mean paper – Avg mean digital =1.18 paper Avg mean paper – Avg mean digital =1.18 paper (slight advantage)(slight advantage)

t=1.78 p=.0776t=1.78 p=.0776

5252

Recap of ResultsRecap of Results Both paper & digital checklist errors were reduced or Both paper & digital checklist errors were reduced or

eliminated during the intervention phaseeliminated during the intervention phase Omission errors were eliminated during reversal and probeOmission errors were eliminated during reversal and probe Timing errors for both paper and digital segments were Timing errors for both paper and digital segments were

eliminated during reversal and probeeliminated during reversal and probe Performance improvement maintained during reversal for both Performance improvement maintained during reversal for both

checklistschecklists Slight performance decrement during post study probe period Slight performance decrement during post study probe period

for both checklistsfor both checklists Dramatic individual intervention effects and phase changes for Dramatic individual intervention effects and phase changes for

both paper & digital checklistsboth paper & digital checklists Results are consistent for the group of participants over timeResults are consistent for the group of participants over time Differences between paper and digital checklists are not Differences between paper and digital checklists are not

statistically significant (can not reject the null hypothesis)statistically significant (can not reject the null hypothesis)

5353

Possible Confounding VariablesPossible Confounding Variables

These variables could account for These variables could account for variability in pilot performance:variability in pilot performance: Flight Training Device experience levelFlight Training Device experience level Recency of flight experienceRecency of flight experience Recency of flight in aircraft typeRecency of flight in aircraft type Fatigue/stressFatigue/stress Recent practice with paper or digital Recent practice with paper or digital

checklistschecklists

5454

LimitationsLimitations Limited timeline of semester Limited timeline of semester

Transferability to other simulator platformsTransferability to other simulator platforms

Transferability to actual flight trainingTransferability to actual flight training

Partial out intervention components (graph Partial out intervention components (graph vs vocal praise)vs vocal praise)

Only used “normal” workload during trialsOnly used “normal” workload during trials

5555

Future ResearchFuture Research Replicating the current study and increase workload conditions such Replicating the current study and increase workload conditions such

as weather, airport traffic, and other distractions as weather, airport traffic, and other distractions Determining how long term gains in checklist accuracy would Determining how long term gains in checklist accuracy would

continue in the absence of post-flight feedback and praise continue in the absence of post-flight feedback and praise Determining how long term gains in checklist accuracy would Determining how long term gains in checklist accuracy would

continue by modifying the detail of post-flight feedback and praise of continue by modifying the detail of post-flight feedback and praise of checklist itemschecklist items

Replicating the current study and ascertaining whether checklist Replicating the current study and ascertaining whether checklist compliance transfers to actual flight compliance transfers to actual flight

Investigating the nature of the rule changes and whether accurate Investigating the nature of the rule changes and whether accurate checklist use would generalize to actual flight checklist use would generalize to actual flight

Curricular modifications regarding formal checklist instruction and Curricular modifications regarding formal checklist instruction and assessment assessment

William Rantz Western Michigan University Department of Aviation Sciences March 16, 2009

Documents

Transcript of William Rantz Western Michigan University Department of Aviation Sciences March 16, 2009