International Civil Aviation Organization

Approved by the Secretary Generaland published under his authority

Line OperationsSafety Audit (LOSA)

First Edition 2002

Doc 9803AN/761

AMENDMENTS

The issue of amendments is announced regularly in the ICAO Journal and in themonthly Supplement to the Catalogue of ICAO Publications and Audio-visualTraining Aids, which holders of this publication should consult. The space belowis provided to keep a record of such amendments.

RECORD OF AMENDMENTS AND CORRIGENDA

AMENDMENTS CORRIGENDA

No.Date

applicableDate

enteredEntered

by No.Date

of issueDate

enteredEntered

by

(ii)

(iii)

TABLE OF CONTENTS

Page Page

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (v)

Acronyms and Abbreviations . . . . . . . . . . . . . . . . . (vi)

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (vii)

Chapter 1. Basic error management concepts . . 1-1

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 1-11.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . 1-2

Reactive strategies . . . . . . . . . . . . . . . . . . 1-2Combined reactive/proactive strategies . . 1-2Proactive strategies . . . . . . . . . . . . . . . . . 1-4

1.3 A contemporary approach to operational human performance and error . . . . . . . . . . . 1-5

1.4 The role of the organizational culture . . . . 1-71.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7

Chapter 2. Implementing LOSA . . . . . . . . . . . . . 2-1

2.1 History of LOSA . . . . . . . . . . . . . . . . . . . . . 2-12.2 The Threat and Error Management Model. 2-1

Threats and errors defined. . . . . . . . . . . . 2-1Definitions of crew error response . . . . . 2-4Definitions of error outcomes . . . . . . . . . 2-4Undesired Aircraft States . . . . . . . . . . . . 2-4

2.3 LOSA operating characteristics . . . . . . . . . 2-5Observer assignment . . . . . . . . . . . . . . . . 2-7Flight crew participation . . . . . . . . . . . . . 2-7

2.4 How to determine the scope of a LOSA . . 2-72.5 Once the data is collected . . . . . . . . . . . . . . 2-82.6 Writing the report . . . . . . . . . . . . . . . . . . . . 2-82.7 Success factors for LOSA . . . . . . . . . . . . . . 2-8

Chapter 3. LOSA and the safety change process (SCP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3-13.2 A constantly changing scene. . . . . . . . . . . . 3-13.3 One operators example of an SCP . . . . . . 3-2

Chapter 4. How to set up a LOSA US Airways experience . . . . . . . . . . . . . . . . . . . . . . 4-1

4.1 Gathering information . . . . . . . . . . . . . . . . . 4-14.2 Interdepartmental support . . . . . . . . . . . . . . 4-14.3 LOSA steering committee. . . . . . . . . . . . . . 4-1

Safety department . . . . . . . . . . . . . . . . . . 4-1Flight operations and trainingdepartments . . . . . . . . . . . . . . . . . . . . . . . 4-2Pilots union . . . . . . . . . . . . . . . . . . . . . . . 4-2

4.4 The key steps of a LOSA . . . . . . . . . . . . . . 4-2Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2Action plan . . . . . . . . . . . . . . . . . . . . . . . 4-2

4.5 The keys to an effective LOSA . . . . . . . . . 4-4Confidentiality and no-jeopardy . . . . . . . 4-4The role of the observer . . . . . . . . . . . . . 4-5

4.6 Promoting LOSA for flight crews . . . . . . . 4-5

Appendix A Examples of the various forms utilized by LOSA . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

Appendix B Example of an introductory letter by an airline to its flight crews . . . . . . . . . . . . . . . . B-1

Appendix C List of recommended reading and reference material . . . . . . . . . . . . . . . . . . . . . . . C-1

(v)

FOREWORD

The safety of civil aviation is the major objective of theInternational Civil Aviation Organization (ICAO). Consider-able progress has been made in increasing safety, butadditional improvements are needed and can be achieved. Ithas long been known that the majority of aviation accidentsand incidents result from less than optimum human per-formance, indicating that any advance in this field can beexpected to have a significant impact on the improvementof aviation safety.

This was recognized by the ICAO Assembly, which in 1986adopted Resolution A26-9 on Flight Safety and HumanFactors. As a follow-up to the Assembly Resolution, the AirNavigation Commission formulated the following objectivefor the task:

To improve safety in aviation by making States more awareand responsive to the importance of Human Factors in civilaviation operations through the provision of practicalHuman Factors materials and measures, developed on thebasis of experience in States, and by developing andrecommending appropriate amendments to existing materialin Annexes and other documents with regard to the role ofHuman Factors in the present and future operationalenvironments. Special emphasis will be directed to theHuman Factors issues that may influence the design,transition and in-service use of the future ICAO CNS/ATMsystems.

One of the methods chosen to implement AssemblyResolution A26-9 is the publication of guidance materials,including manuals and a series of digests, that addressvarious aspects of Human Factors and its impact on aviationsafety. These documents are intended primarily for use byStates to increase the awareness of their personnel of theinfluence of human performance on safety.

The target audience of Human Factors manuals and digestsare the managers of both civil aviation administrations andthe airline industry, including airline safety, training andoperational managers. The target audience also includesregulatory bodies, safety and investigation agencies andtraining establishments, as well as senior and middle non-operational airline management.

This manual is an introduction to the latest informationavailable to the international civil aviation community on thecontrol of human error and the development of counter-measures to error in operational environments. Its targetaudience includes senior safety, training and operationalpersonnel in industry and regulatory bodies.

This manual is intended as a living document and will bekept up to date by periodic amendments. Subsequenteditions will be published as new research results inincreased knowledge on Human Factors strategies and moreexperience is gained regarding the control and managementof human error in operational environments.

(vi)

ACRONYMS AND ABBREVIATIONS

ADS Automatic Dependent SurveillanceATC Air Traffic ControlCFIT Controlled Flight Into TerrainCNS/ATM Communications, Navigation and Surveillance/Air Traffic ManagementCPDLC Controller-Pilot Data Link CommunicationsCRM Crew Resource ManagementDFDR Digital Flight Data RecorderETOPS Extended Range Operations by Twin-engined AeroplanesFAA Federal Aviation AdministrationFDA Flight Data AnalysisFMS Flight Management SystemFOQA Flight Operations Quality AssuranceICAO International Civil Aviation OrganizationLOSA Line Operations Safety AuditMCP Mode Control PanelQAR Quick Access RecorderRTO Rejected Take-OffSCP Safety Change ProcessSOPs Standard Operating ProceduresTEM Threat and Error ManagementUTTEM University of Texas Threat and Error Management

INTRODUCTION

1. This manual describes a programme for themanagement of human error in aviation operations knownas Line Operations Safety Audit (LOSA). LOSA is proposedas a critical organizational strategy aimed at developingcountermeasures to operational errors. It is an organizationaltool used to identify threats to aviation safety, minimize therisks such threats may generate and implement measures tomanage human error in operational contexts. LOSA enablesoperators to assess their level of resilience to systemicthreats, operational risks and front-line personnel errors,thus providing a principled, data-driven approach toprioritize and implement actions to enhance safety.

2. LOSA uses expert and highly trained observers tocollect data about flight crew behaviour and situationalfactors on normal flights. The audits are conducted understrict no-jeopardy conditions; therefore, flight crews are notheld accountable for their actions and errors that areobserved. During flights that are being audited, observersrecord and code potential threats to safety; how the threatsare addressed; the errors such threats generate; how flightcrews manage these errors; and specific behaviours that havebeen known to be associated with accidents and incidents.

3. LOSA is closely linked with Crew ResourceManagement (CRM) training. Since CRM is essentiallyerror management training for operational personnel, datafrom LOSA form the basis for contemporary CRM trainingrefocus and/or design known as Threat and Error Man-agement (TEM) training. Data from LOSA also provide areal-time picture of system operations that can guideorganizational strategies in regard to safety, training andoperations. A particular strength of LOSA is that it identifiesexamples of superior performance that can be reinforced andused as models for training. In this way, training inter-ventions can be reshaped and reinforced based on successfulperformance, that is to say, positive feedback. This is indeeda first in aviation, since the industry has traditionallycollected information on failed human performance, such asin accidents and incidents. Data collected through LOSA areproactive and can be immediately used to prevent adverseevents.

4. LOSA is a mature concept, yet a young one. LOSA

Cathay City, Hong Kong, from 12 to 14 March 2001.Although initially developed for the flight deck sector, thereis no reason why the methodology could not be applied toother aviation operational sectors, including air trafficcontrol, maintenance, cabin crew and dispatch.

5. The initial research and project definition was a jointendeavour between The University of Texas at AustinHuman Factors Research Project and Continental Airlines,with funding provided by the Federal Aviation Admin-istration (FAA). In 1999, ICAO endorsed LOSA as theprimary tool to develop countermeasures to human error inaviation operations, developed an operational partnershipwith The University of Texas at Austin and ContinentalAirlines, and made LOSA the central focus of its FlightSafety and Human Factors Programme for the period 2000to 2004.

6. As of February 2002, the LOSA archives containedobservations from over 2 000 flights. These observationswere conducted within the United States and internationallyand involved four United States and four non-United Statesoperators. The number of operators joining LOSA hasconstantly increased since March 2001 and includes majorinternational operators from different parts of the world anddiverse cultures.

7. ICAO acts as an enabling partner in the LOSAprogramme. ICAOs role includes promoting the importanceof LOSA to the international civil aviation community;facilitating research in order to collect necessary data; actingas a cultural mediator in the unavoidably sensitive aspectsof data collection; and contributing multicultural obser-vations to the LOSA archives. In line with these objectives,the publication of this manual is a first step at providinginformation and, therefore, at increasing awareness withinthe international civil aviation community about LOSA.

8. This manual is an introduction to the concept,methodology and tools of LOSA and to the potentialremedial actions to be undertaken based on the datacollected under LOSA. A very important caveat must beintroduced at this point: this manual is not intended toconvert readers into instant expert observers and/or LOSA(vii)

was first operationally deployed following the First LOSAWeek, which was hosted by Cathay Pacific Airways in

auditors. In fact, it is strongly recommended that LOSA notbe attempted without a formal introduction to it for the

(viii) Line Operations Safety Audit (LOSA)following reasons. First, the forms presented in Appendix Aare for illustration purposes exclusively, since they areperiodically amended on the basis of experience gained andfeedback obtained from continuing audits. Second, formaltraining in the methodology, in the use of LOSA tools and,most important, in the handling of the highly sensitive datacollected by the audits is absolutely essential. Third, theproper structuring of the data obtained from the audits is ofparamount importance.

9. Therefore, until extensive airline experience isaccumulated, it is highly desirable that LOSA training becoordinated through ICAO or the founding partners of theLOSA project. As the methodology evolves and reaches fullmaturity and broader industry partnerships are developed,LOSA will be available without restrictions to theinternational civil aviation community.

10. This manual is designed as follows:

Chapter 1 includes an overview on safety, andhuman error and its management in aviationoperations. It provides the necessary backgroundinformation to understand the rationale for LOSA.

Chapter 2 discusses the LOSA methodology andprovides a guide to the implementation of LOSAwithin an airline. It also introduces a model of crewerror management and proposes the error classi-fication utilized by LOSA, which is essentiallyoperational and practical.

Chapter 3 discusses the safety change process thatshould take place following the implementation ofLOSA.

Chapter 4 introduces the example of one operatorsexperience in starting a LOSA.

Appendix A provides examples of the various formsutilized by LOSA.

Appendix B provides an example of an introductoryletter by an airline to its flight crews.

Appendix C provides a list of recommended readingand reference material.

11. This manual is a companion document to theHuman Factors Training Manual (Doc 9683). Thecooperation of the following organizations in the productionof this manual is acknowledged: The University of Texas atAustin Human Factors Research Project, ContinentalAirlines, US Airways and ALPA, International. Specialrecognition is given to Professor Robert L. Helmreich,James Klinect and John Wilhelm of The University ofTexas at Austin Human Factors Research Project; CaptainsBruce Tesmer and Donald Gunther of Continental Airlines;Captains Ron Thomas and Corkey Romeo of US Airways;and Captain Robert L. Sumwalt III of US Airways and ofALPA, International.

Chapter 1BASIC ERROR MANAGEMENT CONCEPTS

1.1 INTRODUCTION

1.1.1 Historically, the way the aviation industry hasinvestigated the impact of human performance on aviationsafety has been through the retrospective analyses of thoseactions by operational personnel which led to rare anddrastic failures. The conventional investigative approach isfor investigators to trace back an event under considerationto a point where they discover particular actions or decisionsby operational personnel that did not produce the intendedresults and, at such point, conclude human error as the cause.The weakness in this approach is that the conclusion isgenerally formulated with a focus on the outcome, withlimited consideration of the processes that led up to it. Whenanalysing accidents and incidents, investigators alreadyknow that the actions or decisions by operational personnel

are a matter of record. In other words, investigatorsexamining human performance in safety occurrences enjoythe benefit of hindsight. This is, however, a benefit thatoperational personnel involved in accidents and incidentsdid not have when they selected what they thought of asgood or appropriate actions or decisions that would leadto good outcomes.

1.1.2 It is inherent to traditional approaches to safetyto consider that, in aviation, safety comes first. In line withthis, decision making in aviation operations is considered tobe 100 per cent safety-oriented. While highly desirable, thisis hardly realistic. Human decision making in operationalcontexts is a compromise between production and safetygoals (see Figure 1-1). The optimum decisions to achieve theactual production demands of the operational task at handwere bad or inappropriate, because the bad outcomes may not always be fully compatible with the optimum

Safety Production1-1

Figure 1-1. Operational Behaviours Accomplishing the systems goals

1-2 Line Operations Safety Audit (LOSA)decisions to achieve theoretical safety demands. Allproduction systems and aviation is no exception generate a migration of behaviours: due to the need foreconomy and efficiency, people are forced to operate at thelimits of the systems safety space. Human decision makingin operational contexts lies at the intersection of productionand safety and is therefore a compromise. In fact, it mightbe argued that the trademark of experts is not years ofexperience and exposure to aviation operations, but ratherhow effectively they have mastered the necessary skills tomanage the compromise between production and safety.Operational errors are not inherent in a person, although thisis what conventional safety knowledge would have theaviation industry believe. Operational errors occur as a resultof mismanaging or incorrectly assessing task and/or situ-ational factors in a specific context and thus cause a failedcompromise between production and safety goals.

1.1.3 The compromise between production and safetyis a complex and delicate balance. Humans are generallyvery effective in applying the right mechanisms tosuccessfully achieve this balance, hence the extraordinarysafety record of aviation. Humans do, however, occasionallymismanage or incorrectly assess task and/or situationalfactors and fail in balancing the compromise, thuscontributing to safety breakdowns. Successful compromisesfar outnumber failed ones; therefore, in order to understandhuman performance in context, the industry needs tosystematically capture the mechanisms underlying suc-cessful compromises when operating at the limits of thesystem, rather than those that failed. It is suggested thatunderstanding the human contribution to successes andfailures in aviation can be better achieved by monitoringnormal operations, rather than accidents and incidents. TheLine Operations Safety Audit (LOSA) is the vehicleendorsed by ICAO to monitor normal operations.

1.2 BACKGROUND

Reactive strategies

Accident investigation

1.2.1 The tool most often used in aviation to documentand understand human performance and define remedialstrategies is the investigation of accidents. However, in termsof human performance, accidents yield data that are mostlyabout actions and decisions that failed to achieve thesuccessful compromise between production and safetydiscussed earlier in this chapter.

1.2.2 There are limitations to the lessons learned fromaccidents that might be applied to remedial strategies vis--vis human performance. For example, it might be possibleto identify generic accident-inducing scenarios such asControlled Flight Into Terrain (CFIT), Rejected Take-Off(RTO), runway incursions and approach-and-landing acci-dents. Also, it might be possible to identify the type andfrequency of external manifestations of errors in thesegeneric accident-inducing scenarios or discover specifictraining deficiencies that are particularly related to identifiederrors. This, however, provides only a tip-of-the-icebergperspective. Accident investigation, by definition, concen-trates on failures, and in following the rationale advocatedby LOSA, it is necessary to better understand the successstories to see if they can be incorporated as part of remedialstrategies.

1.2.3 This is not to say that there is no clear role foraccident investigation within the safety process. Accidentinvestigation remains the vehicle to uncover unanticipatedfailures in technology or bizarre events, rare as they may be.Accident investigation also provides a framework: if onlynormal operations were monitored, defining unsafebehaviours would be a task without a frame of reference.Therefore, properly focused accident investigation canreveal how specific behaviours can combine with specificcircumstances to generate unstable and likely catastrophicscenarios. This requires a contemporary approach to theinvestigation: should accident investigation be restricted tothe retrospective analyses discussed earlier, its contributionin terms of human error would be to increase existingindustry databases, but its usefulness in regard to safetywould be dubious. In addition, the information couldpossibly provide the foundations for legal action and theallocation of blame and punishment.

Combined reactive/proactive strategies

Incident investigation

1.2.4 A tool that the aviation industry has increasinglyused to obtain information on operational human perform-ance is incident reporting. Incidents tell a more completestory about system safety than accidents do because theysignal weaknesses within the overall system before thesystem breaks down. In addition, it is accepted that incidentsare precursors of accidents and that N-number of incidentsof one kind take place before an accident of the same kindeventually occurs. The basis for this can be traced backalmost 30 years to research on accidents from differentindustries, and there is ample practical evidence thatsupports this research. There are, nevertheless, limitations

Chapter 1. Basic error management concepts 1-3on the value of the information on operational humanperformance obtained from incident reporting.

1.2.5 First, reports of incidents are submitted in thejargon of aviation and, therefore, capture only the externalmanifestations of errors (for example, misunderstood afrequency, busted an altitude, and misinterpreted aclearance). Furthermore, incidents are reported by theindividuals involved, and because of biases, the reportedprocesses or mechanisms underlying errors may or may notreflect reality. This means that incident-reporting systemstake human error at face value, and, therefore, analysts areleft with two tasks. First, they must examine the reportedprocesses or mechanisms leading up to the errors andestablish whether such processes or mechanisms did indeedunderlie the manifested errors. Then, based on this relativelyweak basis, they must evaluate whether the error manage-ment techniques reportedly used by operational personneldid indeed prevent the escalation of errors into a systembreakdown.

1.2.6 Second, and most important, incident reporting isvulnerable to what has been called normalization ofdeviance. Over time, operational personnel develop infor-mal and spontaneous group practices and shortcuts tocircumvent deficiencies in equipment design, clumsy pro-cedures or policies that are incompatible with the realitiesof daily operations, all of which complicate operationaltasks. These informal practices are the product of thecollective know-how and hands-on expertise of a group, andthey eventually become normal practices. This does not,however, negate the fact that they are deviations fromprocedures that are established and sanctioned by theorganization, hence the term normalization of deviance. Inmost cases normalized deviance is effective, at leasttemporarily. However, it runs counter to the practices uponwhich system operation is predicated. In this sense, like anyshortcut to standard procedures, normalized deviance carriesthe potential for unanticipated downsides that mightunexpectedly trigger unsafe situations. However, since theyare normal, it stands to reason that neither these practicesnor their downsides will be recorded in incident reports.

1.2.7 Normalized deviance is further compounded bythe fact that even the most willing reporters may not be ableto fully appreciate what are indeed reportable events. Ifoperational personnel are continuously exposed to sub-standard managerial practices, poor working conditionsand/or flawed equipment, how could they recognize suchfactors as reportable problems?

1.2.8 Thus, incident reporting cannot completelyreveal the human contribution to successes or failures inaviation and how remedial strategies can be improved to

enhance human performance. Incident reporting systems arecertainly better than accident investigations in understandingsystem performance, but the real challenge lies in taking thenext step understanding the processes underlying humanerror rather than taking errors at face value. It is essentialto move beyond the visible manifestations of error whendesigning remedial strategies. If the aviation industry is tobe successful in modifying system and individual per-formance, errors must be considered as symptoms thatsuggest where to look further. In order to understand themechanisms underlying errors in operational environments,flaws in system performance captured through incidentreporting should be considered as symptoms of mismatchesat deeper layers of the system. These mismatches might bedeficiencies in training systems, flawed person/technologyinterfaces, poorly designed procedures, corporate pressures,poor safety culture, etc. The value of the data generated byincident reporting systems lies in the early warning aboutareas of concern, but such data do not capture the concernsthemselves.

Training

1.2.9 The observation of training behaviours (duringflight crew simulator training, for example) is another toolthat is highly valued by the aviation industry to understandoperational human performance. However, the productioncomponent of operational decision making does not existunder training conditions. While operational behavioursduring line operations are a compromise between productionand safety objectives, training behaviours are absolutelybiased towards safety. In simpler terms, the compromisebetween production and safety is not a factor in decisionmaking during training (see Figure 1-2). Trainingbehaviours are by the book.

1.2.10 Therefore, behaviours under monitoredconditions, such as during training or line checks, mayprovide an approximation to the way operational personnelbehave when unmonitored. These observations maycontribute to flesh out major operational questions such assignificant procedural problems. However, it would beincorrect and perhaps risky to assume that observingpersonnel during training would provide the key tounderstanding human error and decision making inunmonitored operational contexts.

Surveys

1.2.11 Surveys completed by operational personnelcan also provide important diagnostic information aboutdaily operations and, therefore, human error. Surveys

1-4 Line Operations Safety Audit (LOSA)Figure 1-2. Training Behaviours Accomplishing training goals

SafetyProductionprovide an inexpensive mechanism to obtain significantinformation regarding many aspects of the organization,including the perceptions and opinions of operationalpersonnel; the relevance of training to line operations; thelevel of teamwork and cooperation among various employeegroups; problem areas or bottlenecks in daily operations;and eventual areas of dissatisfaction. Surveys can also probethe safety culture; for example, do personnel know theproper channels for reporting safety concerns and are theyconfident that the organization will act on expressedconcerns? Finally, surveys can identify areas of dissent orconfusion, for example, diversity in beliefs among particulargroups from the same organization regarding the appropriateuse of procedures or tools. On the minus side, surveyslargely reflect perceptions. Surveys can be likened toincident reporting and are therefore subject to theshortcomings inherent to reporting systems in terms ofunderstanding operational human performance and error.

Flight data recording

1.2.12 Digital Flight Data Recorder (DFDR) andQuick Access Recorder (QAR) information from normalflights is also a valuable diagnostic tool. There are, however,some limitations about the data acquired through these

systems. DFDR/QAR readouts provide information on thefrequency of exceedences and the locations where theyoccur, but the readouts do not provide information on thehuman behaviours that were precursors of the events. WhileDFDR/QAR data track potential systemic problems, pilotreports are still necessary to provide the context withinwhich the problems can be fully diagnosed.

1.2.13 Nevertheless, DFDR/QAR data hold highcost/efficiency ratio potential. Although probably under-utilized because of cost considerations as well as culturaland legal reasons, DFDR/QAR data can assist in identifyingoperational contexts within which migration of behaviourstowards the limits of the system takes place.

Proactive strategies

Normal line operations monitoring

1.2.14 The approach proposed in this manual toidentify the successful human performance mechanisms thatcontribute to aviation safety and, therefore, to the design ofcountermeasures against human error focuses on themonitoring of normal line operations.

Chapter 1. Basic error management concepts 1-51.2.15 Any typical routine flight a normal process involves inevitable, yet mostly inconsequential errors(selecting wrong frequencies, dialling wrong altitudes,acknowledging incorrect read-backs, mishandling switchesand levers, etc.) Some errors are due to flaws in humanperformance while others are fostered by systemic short-comings; most are a combination of both. The majority ofthese errors have no negative consequences because oper-ational personnel employ successful coping strategies andsystem defences act as a containment net. In order to designremedial strategies, the aviation industry must learn aboutthese successful strategies and defences, rather than continueto focus on failures, as it has historically done.

1.2.16 A medical analogy may be helpful inillustrating the rationale behind LOSA. Human error couldbe compared to a fever: an indication of an illness but notits cause. It marks the beginning rather than the end of thediagnostic process. Periodic monitoring of routine flights istherefore like an annual physical: proactively checkinghealth status in an attempt to avoid getting sick. Periodicmonitoring of routine flights indirectly involves measure-ment of all aspects of the system, allowing identification ofareas of strength and areas of potential risk. On the otherhand, incident investigation is like going to the doctor to fixsymptoms of problems; possibly serious, possibly not. Forexample, a broken bone sends a person to the doctor; thedoctor sets the bone but may not consider the root cause(s) weak bones, poor diet, high-risk lifestyle, etc. Therefore,setting the bone is no guarantee that the person will not turnup again the following month with another symptom of thesame root cause. Lastly, accident investigation is like a post-mortem: the examination made after death to determine itscause. The autopsy reveals the nature of a particularpathology but does not provide an indication of theprevalence of the precipitating circumstances. Unfor-tunately, many accident investigations also look for aprimary cause, most often pilot error, and fail to examineorganizational and system factors that set the stage for thebreakdown. Accident investigations are autopsies of thesystem, conducted after the point of no return of the systemshealth has been passed.

1.2.17 There is emerging consensus within the aviationindustry about the need to adopt a positive stance andanticipate, rather than regret, the negative consequences ofhuman error in system safety. This is a sensible objective.The way to achieve it is by pursuing innovative approachesrather than updating or optimizing methods from the past.After more than 50 years of investigating failures andmonitoring accident statistics, the relentless prevalence ofhuman error in aviation safety would seem to indicate asomewhat misplaced emphasis in regard to safety, human

performance and human error, unless it is believed that thehuman condition is beyond hope.

1.3 A CONTEMPORARY APPROACH TOOPERATIONAL HUMAN PERFORMANCE

AND ERROR

1.3.1 The implementation of normal operationsmonitoring requires an adjustment on prevailing views ofhuman error. In the past, safety analyses in aviation haveviewed human error as an undesirable and wrongfulmanifestation of human behaviour. More recently, a con-siderable amount of operationally oriented research, basedon cognitive psychology, has provided a very differentperspective on operational errors. This research has proven,in practical terms, a fundamental concept of cognitivepsychology: error is a normal component of human behav-iour. Regardless of the quantity and quality of regulations theindustry might promulgate, the technology it might design,or the training people might receive, error will continue tobe a factor in operational environments because it simply isthe downside of human cognition. Error is the inevitabledownside of human intelligence; it is the price human beingspay for being able to think on our feet. Practically speak-ing, making errors is a conservation mechanism afforded byhuman cognition to allow humans the flexibility to operateunder demanding conditions for prolonged periods withoutdraining their mental batteries.

1.3.2 There is nothing inherently wrong ortroublesome with error itself as a manifestation of humanbehaviour. The trouble with error in aviation is the fact thatnegative consequences may be generated in operationalcontexts. This is a fundamental point in aviation: if thenegative consequences of an error are caught before theyproduce damage, then the error is inconsequential. Inoperational contexts, errors that are caught in time do notproduce negative consequences and therefore, for practicalpurposes, do not exist. Countermeasures to error, includingtraining interventions, should not be restricted to avoidingerrors, but rather to making them visible and trapping thembefore they produce negative consequences. This is theessence of error management: human error is unavoidablebut manageable.

1.3.3 Error management is at the heart of LOSA andreflects the previous argument. Under LOSA, flaws in humanperformance and the ubiquity of error are taken for granted,and rather than attempting to improve human performance,the objective becomes to improve the context within whichhumans perform. LOSA ultimately aims through changesin design, certification, training, procedures, management

1-8 Line Operations Safety Audit (LOSA)developing analytic methods to integrate multiple anddiverse data sources. However, most importantly, the realchallenge for the large-scale implementation of LOSA willbe overcoming the obstacles, presented by a blame-orientedindustry, that will demand continued effort over time beforenormal operations monitoring is fully accepted by theoperational personnel, whose support is essential.

1.5.2 Despite the challenges and barriers, the aviationsystem has more to gain by moving forward to system-wideimplementation of LOSA than by denying progress becausethat is not the way business has been done in the past or bydecrying the difficulties involved. The following chapterspresent an overview of how to tackle these challenges andbarriers.

Chapter 2IMPLEMENTING LOSA

2.1 HISTORY OF LOSA

2.1.1 In 1991, The University of Texas at AustinHuman Factors Research Project, with funding from theFAA (Human Factors Division, AAR-100), developedLOSA to monitor normal line operations. In its early form,LOSA mostly focused on CRM performance. The reason forthis was that researchers and airlines alike wanted to knowmore about the actual practice of CRM rather than justformulating conclusions about its effectiveness from datacollected within the training environment, as was theestablished practice. After LOSA audits were conducted atmore than ten airlines in the early 1990s, it was clear thatthe actual practice of CRM was quite different than the onedepicted within the typical training department. Mostimportant, the unique insights gathered from this meth-odological approach of monitoring normal operations notonly advanced the concepts of CRM, but also encouragednew ways of thinking about crew performance.

2.1.2 After several years of development andrefinement, LOSA has turned into a strategy of systematicline observations to provide safety data on the way anairlines flight operations system is functioning. The datagenerated from LOSA observations provide diagnosticindicators of organizational strengths and weaknesses inflight operations as well as an overall assessment of crewperformance, both in the technical and human performanceareas. LOSA is a data-driven approach to the developmentof countermeasures to operational threats and errors.

2.2 THE THREAT AND ERRORMANAGEMENT MODEL

2.2.1 LOSA is premised on The University of TexasThreat and Error Management (UTTEM) Model (seeFigure 2-1). Essentially, the model posits that threats anderrors are integral parts of daily flight operations and mustbe managed. Therefore, observing the management or mis-management of threats and errors can build the desired

idea of managing the threats has great relevance to them,more so than error management, which still retains negativeconnotations despite attempts to acknowledge its ubiquityand necessity in human intelligence and informationprocessing. Crew countermeasures are then seen as the toolsthat pilots develop to handle these daily threats and errors.The UTTEM Model has been successfully incorporated intotraining programmes and in some cases has replacedexisting CRM training.*

2.2.2 The UTTEM Model provides a quantifiableframework to collect and categorize data. Some questionsthat can be addressed using this framework include thefollowing:

What type of threats do flight crews most frequentlyencounter? When and where do they occur, and whattypes are the most difficult to manage?

What are the most frequently committed crew errors,and which ones are the most difficult to manage?

What outcomes are associated with mismanagederrors? How many result in an Undesired AircraftState?

Are there significant differences between airports,fleets, routes or phases of flight vis--vis threats anderrors?

2.2.3 The following paragraphs introduce a briefoverview of the most important building blocks of theUTTEM Model.

Threats and errors defined

Threats

2.2.4 Threats are external situations that must bemanaged by the cockpit crew during normal, everydayflights. Such events increase the operational complexity of2-1

systemic snapshot of performance. Pilots quickly grasp theconcepts of external threats once they are explained, and the

* Guidance on Threat and Error Management (TEM) training canbe found in the Human Factors Training Manual (Doc 9683).

2-2 Line Operations Safety Audit (LOSA)

Figure 2-1. The Threat and Error Management Model

Threats

Threat managementInconsequential

Crew error

Crew error responses

UndesiredAircraft State

Crew UndesiredAircraft State

responses

Chapter 2. Implementing LOSA 2-3the flight and pose a safety risk to the flight at some level.Threats may be expected or anticipated and, therefore, thecrew may brief in advance. Threats may also be unexpected.As they occur suddenly and without any warning, there isno possibility for the crew to brief in advance. Externalthreats may be relatively minor or major. Observers shouldrecord all external threats that are on the code sheet or anyothers that may be considered significant.

2.2.5 Errors originated by non-cockpit personnel areconsidered external threats. For example, if the cockpit crewdetects a fuel loading error made by ground staff, it wouldbe entered as an external threat, not an error. The crew wasnot the source of the error (although they must manage it,as they would any other external threat). Other examples ofnon-cockpit crew errors that would be entered as externalthreats are errors in Air Traffic Control (ATC) clearancesdiscovered by the crew, dispatch paperwork errors anddiscrepancies in passenger boarding counts by cabinattendants.

Errors

2.2.6 Cockpit crew error is defined as an action orinaction by the crew that leads to deviations fromorganizational or flight crew intentions or expectations.Errors in the operational context tend to reduce the marginof safety and increase the probability of accidents orincidents. Errors may be defined in terms of non-compliancewith regulations, Standard Operating Procedures (SOPs) andpolicies, or unexpected deviation from crew, company orATC expectations. Errors observed may be minor (selectingthe wrong altitude into the mode control panel (MCP), butcorrecting it quickly) or major (forgetting to do an essentialchecklist). Observers should record all cockpit crew errorsthat they detect.

2.2.7 Operators set up SOPs and checklists as thestandards for the proper and safe way to conduct flights.Instructors observing deviations from SOPs or checklistswould define this as an error, and so does LOSA. If a crewmember does not know how to execute a procedure properlyor cannot control the aircraft in the expected manner, aninstructor would also consider this an error, and so doesLOSA. Deviations from expectations of ATC are alsoclassified as crew errors; these would, for example, includealtitude deviations or significant deviations around thunder-storms without ATC notification. There are rules in SOPsand/or operator manuals that, for example, specify howmuch deviation crews may make around thunderstormsbefore notifying ATC, and observers must be familiar withand apply these company rules when conducting obser-vations. Operators also have policies that are less

proscriptive than procedures, where preferred modes ofoperation are described. Pilots may violate policies withoutviolating SOPs or increasing risk, and under LOSA, this isnot defined as an error. However, if the observer feels thatviolating a policy unnecessarily increases risk to flightsafety, it would be defined as an error. There are also manydecision points on a normal flight that are not defined bySOPs or procedures. However, any time the crew makes adecision that unnecessarily increases risk to flight safety, itis defined as a crew error.

2.2.8 Crew errors may not have any consequences, butthey still need to be recorded by the observer. For example,a violation to the sterile cockpit rule may not have anynegative consequence to the flight, but it is a violation ofregulations and thus must be entered as an error. In addition,errors may be intentional or unintentional. As implied in thedefinition, when a crew action is appropriate or prescribedin SOPs, the lack of action may also be defined as an error.

2.2.9 Is poor crew behaviour that is not a violation ofregulations or SOPs (and did not result in an increased riskto flight safety) deemed an error? For example, shouldobservers enter an error if a crew performed the pre-departure briefing in such a way that it was felt to deservea minimal proficiency? The answer is No. If theminimally proficient or poor pre-departure briefing (or anyother less than optimum behaviour) was not associated withan error of some kind, then it is not an error in its own rightand should not be entered in the observation form.

2.2.10 LOSA is predicated upon the following fivecategories of crew errors:

1. Intentional non-compliance error: Wilful deviationfrom regulations and/or operator procedures;

2. Procedural error: Deviation in the execution ofregulations and/or operator procedures. The inten-tion is correct but the execution is flawed. Thiscategory also includes errors where a crew forgot todo something;

3. Communication error: Miscommunication, mis-interpretation, or failure to communicate pertinentinformation among the flight crew or between theflight crew and an external agent (for example, ATCor ground operations personnel);

4. Proficiency error: Lack of knowledge orpsychomotor (stick and rudder) skills; and

5. Operational decision error: Decision-making errorthat is not standardized by regulations or operator

2-4 Line Operations Safety Audit (LOSA)procedures and that unnecessarily compromisessafety. In order to be categorized as an operationaldecision error, at least one of three conditions musthave existed:

The crew must have had more conservativeoptions within operational reason and decidednot to take them;

The decision was not verbalized and, therefore,was not shared among crew members; or

The crew must have had time but did not use iteffectively to evaluate the decision.

If any of these conditions were observed, then it isconsidered that an operational decision error was made inthe LOSA framework. An example would include the crewsdecision to fly through known wind shear on an approachinstead of going around.

Definitions of crew error response

2.2.11 LOSA considers three possible responses bycrews to errors:

1. Trap: An active flight crew response in which anerror is detected and managed to an inconsequentialoutcome;

2. Exacerbate: A flight crew response in which an erroris detected but the crew action or inaction allows itto induce an additional error, Undesired AircraftState, incident or accident; and

3. Fail to respond: The lack of a flight crew responseto an error because it was either ignored orundetected.

Definitions of error outcomes

2.2.12 The outcome of the error is dependent upon theflight crew response. LOSA considers three possibleoutcomes of errors depending upon crew response:

1. Inconsequential: An outcome that indicates thealleviation of risk that was previously caused by anerror;

2. Undesired Aircraft State: An outcome in which theaircraft is unnecessarily placed in a compromisingsituation that poses an increased risk to safety; and

3. Additional Error: An outcome that was the result ofor is closely linked to a previous error.

Undesired Aircraft States

2.2.13 An Undesired Aircraft State occurs when theflight crew places the aircraft in a situation of unnecessaryrisk. For instance, an altitude deviation is an UndesiredAircraft State that presents unnecessary risk. An UndesiredAircraft State may occur in response to a crew action orinaction (error). It is important to distinguish between errorsand the Undesired Aircraft State that can result. If anUndesired Aircraft State is observed, there should always bea crew error that is responsible for this undesired state. Sucherrors may be miscommunications, lack of proficiency, poordecision making or wilful violation of regulations.

2.2.14 Undesired Aircraft States can also occur as aresult of equipment malfunction or external party errors, forexample, a malfunctioning altimeter or flight managementsystem (FMS), or an ATC command error. These are notassociated with crew error and would be classified asexternal events.

Crew response to Undesired Aircraft States

2.2.15 LOSA considers three possible crew responsesto Undesired Aircraft States:

1. Mitigate: An active flight crew response to anUndesired Aircraft State that results in thealleviation of risk by returning from the UndesiredAircraft State to safe flight;

2. Exacerbate: A flight crew response in which anUndesired Aircraft State is detected, but the flightcrew action or inaction allows it to induce anadditional error, incident or accident; and

3. Fail to respond: The lack of an active flight crewresponse to an Undesired Aircraft State because itwas ignored or undetected.

Definitions of outcomes of Undesired Aircraft States

2.2.16 LOSA considers three possible outcomes toUndesired Aircraft States:

1. Recovery: An outcome that indicates the alleviationof risk that was previously caused by an UndesiredAircraft State;

Chapter 2. Implementing LOSA 2-52. End State/Incident/Accident: Any undesired endingthat completes the activity sequence with a negative,terminal outcome. These outcomes may be of littleconsequence, for example, a long landing or alanding too far to the left or right of the centre line,or may result in a reportable incident or in anaccident; and

3. Additional error: The flight crew action or inactionthat results in or is closely linked to another cockpitcrew error.

2.3 LOSA OPERATING CHARACTERISTICS

2.3.1 LOSA is a proactive safety data collectionprogramme. The data generated provide a diagnosticsnapshot of organizational strengths and weaknesses, as wellas an overall assessment of flight crew performance innormal flight operations. Therefore, the intent of LOSA isto aid airlines in developing data-driven solutions to improveoverall systemic safety. The classic business principle ofmeasure, implement change and measure again is pertinenthere, with LOSA providing the metric of implementationeffectiveness. Experience has proven that expert externaloversight, especially on a first LOSA, is essential forsuccess.

2.3.2 LOSA is defined by the following ten operatingcharacteristics that act to ensure the integrity of the LOSAmethodology and its data. Without these characteristics, it isnot a LOSA. These characteristics are:

1. Jump-seat observations during normal flightoperations: LOSA observations are limited toregularly scheduled flights. Line checks, initial lineindoctrination or other training flights are off-limitsdue to the extra level of stress put upon the pilotsduring these types of situations. Having anotherobserver on board only adds to an already high stresslevel, thus providing an unrealistic picture of per-formance. In order for the data to be representativeof normal operations, LOSA observations must becollected on regular and routine flights.

2. Joint management / pilot sponsorship: In order forLOSA to succeed as a viable safety programme, itis essential that both management and pilots(through their professional association, if it exists)support the project. The joint sponsorship providesa check and balance for the project to ensure thatchange, as necessary, will be made as a result ofLOSA data. When considering whether to conduct

a LOSA audit, the first question to be asked byairline management is whether the pilots endorse theproject. If the answer is No, the project should notbe initiated until endorsement is obtained. This issueis so critical in alleviating pilot suspicion that theexisting LOSA philosophy is to deny airlineassistance if a signed agreement is not in placebefore commencing a LOSA. A LOSA steeringcommittee is formed with representatives from bothgroups and is responsible for planning, scheduling,observer support and, later, data verification (seePoint 8).

3. Voluntary crew participation: Maintaining theintegrity of LOSA within an airline and the industryas a whole is extremely important for long-termsuccess. One way to accomplish this goal is tocollect all observations with voluntary crewparticipation. Before conducting LOSA obser-vations, an observer must first obtain the flightcrews permission to be observed. The crew has theoption to decline, with no questions asked. Theobserver simply approaches another flight crew onanother flight and asks for their permission to beobserved. If an airline conducts a LOSA and has anunreasonably high number of refusals by crews to beobserved, then it should serve as an indicator to theairline that there are critical trust issues to be dealtwith first.

4. De-identified, confidential and safety-minded datacollection: LOSA observers are asked not to recordnames, flight numbers, dates or any otherinformation that can identify a crew. This allows fora level of protection against disciplinary actions. Thepurpose of LOSA is to collect safety data, not topunish pilots. Airlines cannot allow themselves tosquander a unique opportunity to gain insight intotheir operations by having pilots fearful that a LOSAobservation could be used against them fordisciplinary reasons. If a LOSA observation is everused for disciplinary reasons, the acceptance ofLOSA within the airline will most probably be lostforever. Over 6 000 LOSA observations have beenconducted by The University of Texas at AustinHuman Factors Research Project and not one hasever been used to discipline a pilot.

5. Targeted observation instrument: The current datacollection tool to conduct a LOSA is the LOSAObservation Form. It is not critical that an airline usethis form, but whatever data collection instrument isused needs to target issues that affect flight crewperformance in normal operations. An example of

2-6 Line Operations Safety Audit (LOSA)the LOSA Observation Form is shown inAppendix A. The form is based upon the UTTEMModel and generates data for a variety of topics,including the following:

Flight and crew demographics such as city pairs,aircraft type, flight time, years of experiencewithin the airline, years of experience withinposition, and crew familiarity;

Written narratives describing what the crew didwell, what they did poorly and how theymanaged threats or errors for each phase of theflight;

CRM performance ratings using research-developed behavioural markers;

Technical worksheet for the descent/approach/land phases that highlights the type of approachflown, landing runway and whether the crew metairline stabilized approach parameters;

Threat management worksheet that details eachthreat and how it was handled;

Error management worksheet that lists eacherror observed, how each error was handled andthe final outcome; and

Crew interview conducted during low workloadperiods of the flight, such as cruise, that askspilots for their suggestions to improve safety,training, and flight operations.

6. Trusted, trained and calibrated observers: Primarily,pilots conduct LOSAs. Observation teams willtypically include line pilots, instructor pilots, safetypilots, management pilots, members of HumanFactors groups and representatives of the safetycommittee of the pilots organization. Another part ofthe team can include external observers who are notaffiliated with the airline. If they have no affiliationwith the airline, external observers are objective andcan serve as an anchor point for the rest of theobservers. Trained, expert external observers addtremendous value, especially if they have par-ticipated in LOSA projects at other airlines. It iscritical to select observers that are respected andtrusted within the airline to ensure the lines accept-ance of LOSA. Selecting good observers is thelifeline of LOSA. If you have unmotivated oruntrustworthy observers, LOSA will fail. The size ofthe observation team depends on the airlines size,

the number of flights to be observed and the lengthof time needed to conduct the observations. Afterobservers are selected, everyone is trained andcalibrated in the LOSA methodology, including theuse of the LOSA rating forms and, particularly, theconcepts of threat and error management. Trainingof observers in the concepts and methodology ofLOSA will ensure that observations will be con-ducted in the most standardized manner. Aftercompleting training, observers spend a period oftime (between one and two months) observingregularly scheduled line flights. The objective is toobserve the largest number of crews and segmentspossible in the time frame, given the flightschedules, logistics and types of operation sampled.

7. Trusted data collection site: In order to maintainconfidentiality, airlines must have a trusted datacollection site. At the present time, all observationsare sent off-site directly to The University of Texasat Austin Human Factors Research Project, whichmanages the LOSA archives. This ensures that noindividual observations will be misplaced orimproperly disseminated through the airline.

8. Data verification roundtables: Data-driven pro-grammes like LOSA require quality data man-agement procedures and consistency checks. ForLOSA, these checks are done at data verificationroundtables. A roundtable consists of three or fourdepartment and pilots association representativeswho scan the raw data for inaccuracies. For example,an observer might log a procedural error for failureto make an approach callout for which there areactually no written procedures in the airlines flightoperations manual. Therefore, it would be the job ofthe roundtable to detect and delete this particularerror from the database. The end product is adatabase that is validated for consistency andaccuracy according to the airlines standards andmanuals, before any statistical analysis is performed.

9. Data-derived targets for enhancement: The finalproduct of a LOSA is the data-derived LOSAtargets for enhancement. As the data are collectedand analysed, patterns emerge. Certain errors occurmore frequently than others, certain airports orevents emerge as more problematic than others,certain SOPs are routinely ignored or modified andcertain manoeuvres pose greater difficulty in ad-herence than others. These patterns are identified forthe airline as LOSA targets for enhancement. It isthen up to the airline to develop an action plan basedon these targets, using experts from within the airline

Chapter 2. Implementing LOSA 2-7to analyse the targets and implement appropriatechange strategies. After two or three years, theairline can conduct another LOSA to see if theirimplementations to the targets show performanceimprovements.

10. Feedback of results to the line pilots: After a LOSAis completed, the airlines management team andpilots association have an obligation to communicateLOSA results to the line pilots. Pilots will want tosee not only the results but also managements planfor improvement. If results are fed back in an appro-priate fashion, experience has shown that futureLOSA implementations are welcomed by pilots andthus more successful.

2.3.3 Over the years of implementation, the tenoperating characteristics listed above have come to defineLOSA. Whether an airline uses third party facilitation orattempts to do a LOSA by itself, it is highly recommendedthat all ten characteristics are present in the process. Overthe past five years, the most valuable lesson learned was thatthe success of LOSA goes much beyond the data collectionforms. It depends upon how the project is executed andperceived by the line pilots. If LOSA does not have the trustfrom the pilot group, it will probably be a wasted exercisefor the airline.

Observer assignment

2.3.4 Members of the observation teams are typicallyrequired to observe flights on different aircraft types. Thisis an important element of the line audit process for severalreasons. For one, this has the advantage of allowing both linepilots and instructor pilots of particular fleets to break outof the box (their own fleet) and compare operations of fleetsother than their own. Eventually, this helps the team as awhole to focus on Human Factors issues and commonsystemic problems, rather than on specific, within-fleetproblems. Furthermore, the results are more robust ifobservers observe across many fleets instead of observingonly one type.

Flight crew participation

2.3.5 Normally the line audit is announced to crewmembers by means of a letter from the highest level ofmanagement within flight operations, with the endorsementof other relevant personnel such as chief pilots and pilotsassociation representatives. This letter specifies the purposeof the audit and the fact that all observations are of a no-jeopardy nature and all data are to be kept strictly

confidential. The letter of announcement should precede theline audit by at least two weeks, and line observers are givencopies of the letter to show crew members in case questionsshould arise. Data are kept anonymous and crews are givenassurance that they are not in disciplinary jeopardy.Furthermore, crews should have the option to refuseadmission of the observer to perform an observation on theirflight.

2.4 HOW TO DETERMINE THESCOPE OF A LOSA

2.4.1 Only smaller airlines with limited numbers offleets would find it reasonable to attempt to audit their entireflight operation, that is, all types of operations and all fleets.Most airlines will find it cost effective to conduct a LOSAon only parts of their operation. Evidence from LOSAsuggests that flight crew practices vary naturally by fleet.The type of operation, such as domestic, international, short-haul or long-haul, is also relevant. Usually, auditing anycombination of types of operations is a good way to breakdown an entire operation into useful comparison groups.

2.4.2 Ideally, every flight crew should be audited, butmore often than not, this will be impossible or impracticalin material terms. At a major airline and in large fleets,around 50 randomly selected flight crews will providestatistically valid data. For smaller fleets, around 30 ran-domly selected flight crews will provide statistically validdata, although the risk of arriving at conclusions that mightnot reflect reality increases as the number of flight crewsaudited drops. If less than 25 flight crews are audited, thedata collected should be considered as case studies ratherthan representing the group as a whole.

2.4.3 The number of observers needed depends, asalready discussed, on the intended scope of the audit. Forexample, an airline might want to audit 50 flight crews ineach of 2 domestic fleets, for a total of 100 segments. Aconservative rule of thumb to scope this audit would be2 domestic observations per day per observer. The goal isthus expressed in terms of flight crews observed, rather thansegments. Should an airline want to audit an internationalfleet, the first step is to determine how many internationalobservations can be made in a day, and this depends on thelength of the segments. For a domestic LOSA, a workablerule of thumb suggests the need for 50 person/days of workfor the actual audit phase of the LOSA. Using line pilots fora month of observations, each might be requested to spend10 days conducting observations, plus 4 days training/travelling. This requires 14 days per observer. Thus, therewould be a need for 4 observers for this hypothetical audit,

2-8 Line Operations Safety Audit (LOSA)and this should easily meet the audits goals. It is importantto be conservative in the estimates since sometimes it willbe necessary to observe a crew for more than one segment.This counts as one crew, not two.

2.5 ONCE THE DATA IS COLLECTED

The data acquired through the observations must beverified and prepared for analysis, and the time involvedin this process should not be underestimated. Once thevarious LOSA forms have been collected, the airline is readyto begin a lengthy process. It typically takes longer toprepare the LOSA data for analysis and ulterior action thanit does to collect it. The steps that must be followed in thisprocess include data entry, data quality/consistency checksand final aggregation.

2.6 WRITING THE REPORT

2.6.1 The last stage of LOSA is a written report thatpresents the overall findings of the project. With a largedatabase like the one generated from a LOSA, it is easy tofall into the trap of trying to present too much information.The author needs to be concise and present only the mostsignificant trends from the data. If the report does notprovide a clear diagnosis of the weaknesses within thesystem for management to act upon, the objective of theLOSA will be unfulfilled.

2.6.2 Writing the report is where data smarts entersinto the process. Although certain types of comparisons willseem obvious, many analyses will be based upon thehunches or theories of the writer. The usefulness of theresult has to be the guiding principle of this effort. If thewriter knows how fleets and operations are managed,comparisons that reflect this structure can be made. If theauthor knows the kinds of information that might be usefulto training, safety or domestic/international flight oper-ations, results can be tailored to these particular aspects ofthe operation. Feedback from various airline stakeholders iscritical during this stage of writing the report. Authorsshould not hesitate to distribute early drafts to key peoplefamiliar with LOSA to verify the results. This not only helpsvalidate derived trends, but it gives other airline personnel,besides the author, ownership of the report.

2.6.3 General findings from the survey, interview andobservational data should serve as the foundation inorganizing the final report. A suggested outline for the reportfollows:

Introduction Define LOSA and the reasons why itwas conducted.

Executive Summary Include a text summary of themajor LOSA findings (no longer than two pages).

Section Summaries Present the key findings from eachsection of the report including:

I DemographicsII Safety Interview ResultsIII External Threats and Threat Management

ResultsIV Flight Crew Errors and Error Management

ResultsV Threat and Error Countermeasure Results

Appendix Include a listing of every external threatand flight crew error observed with the proper codingand an observer narrative of how each one was managedor mismanaged.

Tables, charts and explanations of data should be providedwithin each section of the report.

2.6.4 It is important to remember that the authorsprimary job is to present the facts and abstain from outliningrecommendations. This keeps the report concise andobjective. Recommendations and solutions may be givenlater in supporting documentation after everyone has had thechance to digest the findings.

2.7 SUCCESS FACTORS FOR LOSA

The best results are achieved when LOSA is conducted inan open environment of trust. Line pilots must believe thatthere will be no repercussions at the individual level;otherwise, their behaviour will not reflect daily operationalreality and LOSA will be little more than an elaborate linecheck. Experience at different airlines has shown that severalstrategies are key to ensuring a successful, data-rich LOSA.These strategies include:

Using third-party oversight: One way to build trustin the LOSA process is to seek a credible but neutralthird party who is removed from the politics andhistory of the airline. Data can be sent directly to thisthird party, who is then responsible for the objectiveanalyses and report preparation. The University ofTexas at Austin Human Factors Research Projectprovides, for the time being, such third partyoversight;

Chapter 2. Implementing LOSA 2-9 Promoting LOSA: Use group presentations, mediaclippings, experience from other airlines and intra-airline communications to discuss the purpose andlogistics of a LOSA audit with management, pilotsand any pilots associations. Experience shows thatairlines often underestimate the amount of com-munication required so they must be persistent intheir efforts;

Stressing that observations cannot be used fordiscipline purposes: This is the key issue and mustbe stated as such in the letter of endorsement;

Informing the regulatory authority of the proposedactivity: It is as much a courtesy as it is a way ofcommunicating the presence of LOSA;

Choosing a credible observer team: A line crewalways has the prerogative to deny cockpit access toan observer; hence the observer team is mosteffective when composed of credible and well-accepted pilots from a mix of fleets and departments(for example, training and safety). This was achievedat one airline by asking for a list of potentialobservers from the management and the pilots

association; those pilots whose names appeared onboth lists were then selected as acceptable toeveryone;

Using a fly on the wall approach: The bestobservers learn to be unobtrusive and non-threatening; they use a pocket notebook while in thecockpit, recording minimal detail to elaborate uponlater. At the same time, they know when it isappropriate to speak up if they have a concern,without sounding authoritarian;

Communicating the results: Do not wait too longbefore announcing the results to the line or elsepilots will believe nothing is being done. A summaryof the audit, excerpts from the report and relevantstatistics will all be of interest to the line; and

Using the data: The LOSA audit generates targetsfor enhancement, but it is the airline that creates anaction plan. One airline did this by creating acommittee for each of the central concerns, and theywere then responsible for reviewing procedures,checklists, etc., and implementing change, whereappropriate.

Chapter 3LOSA AND THE SAFETY CHANGE PROCESS (SCP)

3.1 INTRODUCTION

3.1.1 When an airline commits to LOSA, it must alsocommit to acting upon the results of the audit. LOSA is buta data collection tool. LOSA data, when analysed, are usedto support changes aimed at improving safety. These may bechanges to procedures, policies or operational philosophy.The changes may affect multiple sectors of the organizationthat support flight operations. It is essential that the organ-ization has a defined process to effectively use the analyseddata and to manage the changes the data suggest.

3.1.2 LOSA data should be presented to managementin at least the areas of operations, training, standards andsafety, with a clear analysis describing the problems relatedto each, as captured by LOSA. The LOSA report shouldclearly describe the problems the analysed data suggest butshould not attempt to provide solutions. These will be betterprovided through the expertise in each of the areas inquestion.

3.1.3 LOSA directs organizational attention to themost important safety issues in daily operations and itsuggests what questions should be asked; however, LOSAdoes not provide the solutions. The solutions lie inorganizational strategies. The organization must evaluate thedata obtained through LOSA, extract the appropriateinformation and then carry out the necessary interventionsto address the problems thus identified. LOSA will onlyrealize its full potential if the organizational willingness andcommitment exist to act upon the data collected and theinformation such data support. Without this necessary step,LOSA data will join the vast amounts of untapped dataalready existing throughout the international civil aviationcommunity.

3.1.4 The following would be some of the typicalexpected actions, in no particular order, by an airlinefollowing a LOSA:

Modifying existing procedures or implementing newones;

Arranging specific training in error management andcrew countermeasures;

Reviewing checklists to ensure relevance of thecontent and then issuing clear guidelines for theirinitiation and execution; and

Defining tolerances for stabilized approaches, asopposed to the perfect approach parameterspromoted by existing SOPs.

3.2 A CONSTANTLY CHANGING SCENE

3.2.1 Airlines are continually involved in changes that,at some point or other, have an effect upon safety. Factorsunderlying this continuous change process include, amongothers, regulatory changes (airworthiness directives, advis-ory circulars, etc.); changes in national or internationalairspace systems (automatic dependent surveillance (ADS),data link/controller-pilot data link communications(CPDLC), reduced vertical separation, extended rangeoperations by twin-engined aeroplanes (ETOPS), etc.);changes to improve operational efficiencies (reduction ofcosts, improvement of on-time performance, etc.); opera-tional events (diversions, rejected take-offs, etc.); andprogress (route expansion, fleet modernization, newtechnologies, etc.)

3.2.2 Virtually everyone in an airline is somehowinvolved in these changes. For example, Chief ExecutiveOfficers and their staff decide to buy new equipment;marketing opens up new routes; engineering must installnew components; flight operations faces new staffingrequirements and adjustments to line procedures; flightstandards must define new policies and procedures; andflight training faces acquisition of new simulators.

3.2.3 These changes are achieved and monitored boththrough established formal and informal mechanismsunderlying change processes. Formal mechanisms include3-1

Redefining operational philosophies and guidelines;meetings (daily, weekly, monthly and quarterly), reports andreviews at all levels of the organization. Informal

3-2 Line Operations Safety Audit (LOSA)mechanisms include crosstalk, spontaneous informationtransfer, and sharing in general by everyone in theorganization. Both mechanisms work toward activelymaintaining focus on the changes affecting safety.

3.2.4 Therefore, when in spite of these formal andinformal mechanisms an airline experiences an accident oran incident, the immediate question arises: What ishappening out there? The fact is that system changes andorganizational responses to these changes generate activeand latent threats to daily line operations. Active and latentthreats themselves constantly change in a manner pro-portional to system changes. Active and latent threatsbecome the breeding grounds of crew errors. Many organ-izations are not aware of these active and latent threats fora number of reasons, including the following:

The big picture of flight operations is constantlychanging because of the constantly changing scene;

Crews may not report threats, fearing punishment;

Crews may not report threats because they do notreceive any feedback on their reports;

Crews operate unsupervised most of the time;

Line Checks (supervised performance) are poorindicators of normal operations; and

Management may have difficulty screening out validreported crew concerns from over-reported crewcomplaints.

3.2.5 Active and latent threats are the precursors toaccidents and incidents. Threats cannot be identified throughthe investigation of accidents and incidents until it is too late.Most threats, however, can be proactively identified throughLOSA (and other safety data collection programmes such asflight data analysis) and considered as targets for enhance-ment. For example, following a LOSA, an airline mightidentify the following targets for enhancement:

Stabilized approaches

Checklists

Procedural errors

Automation errors

ATC communications

International flight operations guide

Captain leadership (intentional non-complianceerrors)

3.2.6 To sustain safety in a constantly changingenvironment, data must be collected and analysed on aroutine basis to identify the targets for enhancement and thena formal safety change process (SCP) must occur in orderto bring about improvement. The basic steps of the SCPinclude the following and are also shown in Figure 3-1.

Measurement (with LOSA) to obtain the targets

Detailed analysis of targeted issues

Listing of potential changes for improvement

Risk analysis and prioritization of changes

Selection and funding of changes

Implementation of changes

Time allocation for changes to stabilize

Re-measurement

3.2.7 Airlines need a defined SCP to keep theorganization working together to achieve the same safetyobjectives. A well-defined SCP keeps the organization fromgetting into turf issues, by clearly specifying who andwhat impacts flight operations. An SCP also contributes toimproving the safety culture by maximizing the capabilitiesof current and future safety programmes. Last, but not least,an SCP provides a principled approach to target limitedresources.

3.2.8 In the past, SCPs were based on accident andincident investigations, experience and intuition. Today,SCPs must be based on the data wave, the datawarehouse and the drill-down analysis. Measurement isfundamental, because until an organization measures, it canonly guess. In the past, SCPs dealt with accidents. Today,SCPs must deal with the precursors of accidents.

3.3 ONE OPERATORS EXAMPLEOF AN SCP

3.3.1 This section briefly presents some of the verypositive results obtained by one airline that pioneered LOSAin international civil aviation. The examples represent atwo-year period, between 1996 and 1998, and include

Chapter 3. LOSA and the safety change process (SCP) 3-3Measurementaggregate data collected during 100 flight segments. Duringthis two-year period, 85 per cent of the crews observed madeat least one error during one or more segments, and 15 percent of the crews observed made between two and fiveerrors. Errors were recorded in 74 per cent of the segmentsobserved, with an average of two errors per segment (seeChapter 2 for a description of the error categories in LOSA).These data, asserted as typical of airline operations,substantiated the pervasiveness of human error in aviationoperations, while challenged beyond question the illusion oferror-free operational human performance.

3.3.2 LOSA observations indicated that 85 per cent oferrors committed were inconsequential, which led to twoconclusions. First, the aviation system possesses very strongand effective defences, and LOSA data allow a principledand data-driven judgement of which defences work andwhich do not, and how well defences fulfil their role.Second, it became obvious that pilots intuitively develop adhoc error management skills, and it is therefore essential todiscover what pilots do well so as to promote safety throughorganizational interventions, such as improved training,procedures or design, based on this positive data.

Figure 3-1. Basic steps of the safety change process

Re-measurement

Risk analysis andprioritization of changes

Listing ofpotential changes

Analysis oftargets

Time allocationfor changes

Implementationof changes

Selection andfunding of changes

LOSA

3-4 Line Operations Safety Audit (LOSA)

3.3.3 When the airline started conducting base-lineobservations in 1996, the crew error-trapping rate was 15 percent; that is, flight crews detected and trapped only 15 percent of the errors they committed. After two years, followingimplementation of organizational strategies aimed at errormanagement based on LOSA data, the crew error-trappingrate increased to 55 per cent (see Figure 3-2).

3.3.4 Base-line observations in 1996 suggestedproblems in the area checklist performance. Followingremedial interventions including review of standardoperating procedures, checklist design and training checklist performance errors decreased from 25 per cent to15 per cent, which is a 40 per cent reduction in checklisterrors (see Figure 3-3).

Figure 3-2. Crew error-trapping rate

15%

55%

0%

10%

20%

40%

50%

60%

1996 1998

30%

25%

15%10%

15%

20%

25%40%

ReductionFigure 3-3. Checklist errors

0%

5%

1996 1998

Chapter 3. LOSA and the safety change process (SCP) 3-53.3.5 Lastly, base-line observations in 1996 suggestedthat 34.2 per cent of approaches did not meet all require-ments of the audits stabilized approach criteria, as specifiedin the operators SOPs. Unstabilized approaches (using morestringent criteria than during the 1996 audit) decreased to13.1 per cent (a 62 per cent reduction) in 1998, followingremedial action through organizational interventions. Thedata accessed through the operators flight operations qualityassurance (FOQA) programme is consistent with LOSA dataand shows a similar decline for 1998.

3.3.6 How does such change take place? By adoptinga defined SCP. Following data acquisition and analysis, theairline decided to form specific committees including achecklist committee and an unstabilized approaches com-mittee. Each committee considered the problems identifiedby the analysis of the LOSA data and then proposedorganizational interventions to address them. Such inter-ventions included modification of existing procedures,

implementation of new ones, specific training, andredefinition of operational philosophies, among others. Forexample, checklists were reviewed to ensure relevance ofcontents, and clear guidelines for their initiation andexecution were promulgated. Gates and tolerances forstabilized approaches were defined, as opposed to theperfect approach parameters promulgated by the SOPsexisting at that time. Proper training and checking guidelineswere established, taking into account an error managementapproach to crew coordination.

3.3.7 The improved error management performance byflight crews, successful reduction in checklist performanceerrors and reduction in unstabilized approaches discussedearlier reflect the success of a properly managed SCP, basedupon data collected by observing line operations. They arealso examples of how analysis of LOSA data provides anopportunity to enhance safety and operational humanperformance.

Chapter 4HOW TO SET UP A LOSA US AIRWAYS EXPERIENCE

Honest and critical self-assessment is one of the mostpowerful tools that management can employ to measureflight safety margins.

Flight Safety Foundation Icarus CommitteeMay 1999

4.1 GATHERING INFORMATION

In order to decide if conducting a LOSA would bebeneficial, it is important to understand the LOSA process.The first contact should be ICAO or The University of Texasat Austin Human Factors Research Project. Both are able toprovide all the information needed and can discuss benefitsthat have been derived from past LOSAs. They will also beaware of other airlines currently planning or conducting aLOSA, and it may be possible to attend those airlinestraining classes. It is also a good idea to talk with and/or visitother airlines that have already completed a LOSA to learnfrom their experiences.

4.2 INTERDEPARTMENTAL SUPPORT

4.2.1 When first exploring whether or not to conducta LOSA, it is advisable to gather representatives from alldepartments that may be potentially involved. This couldinclude the flight operations, training and safety depart-ments, as well as representatives from the pilots union. IfLOSA is not supported by all concerned, the effectivenessof LOSA will be compromised.

4.2.2 As an example, a few years ago a large airlinedecided to audit its line flights on a somewhat random basis.The audit was not a LOSA but did have similarities in thatit used trained observers to ride in the airlines cockpits. Theairlines safety department administered the line audit, andthe data that they collected were valid and important.

by the safety department telling them what was wrongwith the airline, and therefore they were not very receptiveto the findings of this particular line audit.

4.2.3 A few years later, this same airline conducted avery successful LOSA. This time, the airline emphasizedthat the audit was not owned by the safety department, butrather, was a product of the flight operations, training andsafety departments, along with the pilots union. Each ofthese departments and organizations became members of theLOSA steering committee. This airlines LOSA wassuccessful for many reasons, but primarily because rightfrom the start, all relevant departments were involved withthe development and direction that the LOSA took. In short,the programme had interdepartmental buy-in.

4.3 LOSA STEERING COMMITTEE

4.3.1 This buy-in and support of other departments arecrucial; therefore, consideration should be given to forminga LOSA steering committee. Determining which depart-ments should be members varies with each organization but,at a minimum, should include the safety, flight operationsand flight training departments and the pilots union. The roleof each of these is described below.

Safety department

4.3.2 Ideally, the safety department should be thedepartment to administer the LOSA. There are several4-1

However, the problem was that the flight operations andtraining departments of this airline felt somewhat threatened

reasons for this. For one, conducting audits is typically a jobfunction of the safety department. Another important reason

4-2 Line Operations Safety Audit (LOSA)is that the safety department often holds the trust of the linepilots regarding confidential information. It is the safetydepartment that typically administers confidential incidentreporting systems and the FOQA Programme or digitalflight data recorder monitoring programmes.

Flight operations and training departments

4.3.3 The flight operations and training departmentsmust be integrally involved with implementing a LOSA forseveral reasons. First, they are at the centre of the operationand have first-hand information about what is and is notworking well. These departments often know of specificareas on which they would like the LOSA to concentrate.Additionally, these departments can provide valuable inputand suggestions for the smooth conduct of the LOSA. Theywill also be able to help provide the much needed personnel.Possibly the most important reason for their involvement isthat ultimately many of the problem areas and the potentialbenefits that are identified during a LOSA must becorrected or implemented by these departments. As withthe example of the airline above, if these departments do notsupport the LOSA, then there could be possible resistanceto the findings from the LOSA. However, if these depart-ments take an active part in the process, implementation ofLOSA enhancements becomes much more probable.

Pilots union

4.3.4 The importance of having the pilots unioninvolved with and support the LOSA must not beoverlooked. If the line pilots believe that their union supportsthis endeavour, they will more readily accept observationflights. Additionally, if pilots believe this is a process thatthey can support, they will be more forthcoming and candidwith their views and safety concerns. On the other hand, ifthe pilots view LOSA as a management tool to spy on them,then the results will not be as productive. The pilots unioncan also help disseminate the results of the LOSA andinform the pilots of any company decisions as a result of theLOSA. Hopefully, the union will agree with theenhancements and endorse them.

4.4 THE KEY STEPS OF A LOSA

4.4.1 To help provide focus for the LOSA, the LOSAsteering committee should first look at problems that havebeen identified in the past by all involved departments. Withthis information in hand, the committee can then decidewhat they expect to gain from the LOSA and use that to form

goals and an action plan. It must be kept in mind that thegoals and action plan may have to be modified dependingon the LOSA findings.

Goals

4.4.2 The LOSA steering committee should meet todetermine what they would like to achieve from the LOSA.This will vary among airlines, but the following are somegoals established by one airline:

To heighten the safety awareness of the line pilot

To obtain hard data on how crews manage threatsand errors

To measure and document what is happening on theline

What works well What does not work well

To provide feedback to the system so thatenhancements can be made

To inform end users WHY enhancements are beingmade, especially if the enhancements are a result ofend user feedback

To monitor results of LOSA enhancements

4.4.3 One airline stated up front that they wanted theirline pilots to be the customer of the LOSA, meaning thatwhatever problems were identified, they would work tocorrect them to make the system safer and more efficient fortheir pilots.

Action plan

4.4.4 Figure 4-1 shows a flow chart of the key stepsto LOSA. Steps 1 to 6 are covered below. Notice that theactual LOSA observations are not the end of the project but,in fact, are only a part of an entire process to help improvesystem safety at an airline. Steps 7 to 9 have already beencovered earlier in this manual.

Step 1: Form initial development team

This team may be the same as the LOSA steering committeeor just a few core individuals who can bring the committeeup to date.

Chapter 4. How to set up a LOSA US Airways experience 4-3

Figure 4-1. The key steps to LOSA

STEP 5: Schedule audit dates,select observers and

schedule training

STEP 8: Provide feedback to systemand carry out improvements to system

STEP 9: Develop enhanced policies,procedures and a safer operational

environment for the line pilot

STEP 4: Determine howmany segments to observe

STEP 6: Conduct observer training

STEP 7: Analyse audit findings

AUDIT

STEP 3: Identify what to lo