Risk Analysis Tool Process (RAT/RAP) Meetings Seminars and Workshops/Safety... · 1 Risk Analysis...

1

Risk Analysis Tool & Process (RAT/RAP)The way the harmonise assessment of ATM incidents over EUROPE and FAA

Tony LICU

Gilles LE GALO

Radu CIOPONEA

Florin CIORAN

EUROCONTROL

Art SHEFFIELD

James C. BEADOW

Jon JONES

Mike McFADYEN

FAA

The European Organisation for the Safety of Air Navigation

Federal AviationAdministration

WORKSHOP ON IMPLEMENTATION OF THE ICAO SAFETY MANAGEMENT REQUIREMENTS IN STATES

Paris, France – 14/15 February 2011

ICAO Annex 13

• First edition September 1951

• 1st‐3rd edition (04/1973) called“Aircraft Accident Inquiry”Aircraft Accident Inquiry

• 4th‐7th edition (05/1988) called“Aircraft Accident Investigation”

• 8th‐9th edition (07/2001) called“Aircraft Accident and IncidentInvestigation”


SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 2

2

Annex 13 First Edition



ATM Safety reporting and investigation evolution in Europe

• ESARR 2 (ATM Occurrence Reporting and Investigation) and the first Risk Classification Scheme ‐ 1999

• Risk Wizard 2004/2005• Risk Wizard – 2004/2005

• RAT – Risk Analysis Tool ‐ 2009

• Implementing rule on Performance EC691/2010



1999 20062004 2007 2009 2010

Safety KPI

Roadmap

SAFREP &

RAT

ESARR 2 &

EAM2GUI1

Risk Wizard IR on Performance

3

European Regulatory environment change

•Need to know what occurs at European level

•States /ANSPs report to EUROCONTROL as of 1999

•EUROCONTROL to analyse the data

•Action Plans

•SESII PRB Performance framework



HARMONISATION

•SESII, PRB, Performance framework

ESARR 2

Need to know about undesired events that have had or might have had an impact on safety

Reporting systems

Requires a common

TAXONOMY

Reported Safety Occurrences

Findings, RecommendationsSeverity & Risk Assessment

Need to determine to what extent ATM has contributed to the occurrences and severity of a safety risk

Data collectionAnalysis

Severity & Severity & Risk assessmentRisk assessmentATM contribution

Requires

HARMONISED PROCESSES



Severity & Risk Assessment

Need to share experiences

Trends, KRA, ATM improvementsGLOBAL SOLUTIONS

Annual Summary Template

Agreements(bilateral orregional)

Requires

PROCEDURES

CONFIDENTIAL ASSURANCE

4

Status in Europe back in 1999

• State shall ensure “that the severity of occurrences is determined, the risk posed by occurrences classified, and the results recorded.” (See ESARR 2 –section 5.1.6)

• The assessment/investigation of the occurrence shall enable the determination of the severity level of the occurrence (ESARR 2‐ A‐3.1)

• EAM 2‐ GUI 1 states Supplementary material will be developed with experience to decrease the level of subjectivity inherent to severity



p j y yassessment

D1 E1 C1 B1 A11

very frequen

tnt

ESARR2 in Details ‐ Classification (Incidents)Risk Classification Scheme for occurrences involving aircraft and their occupants (European version)

D4

D3

D2

E4

E3

E2

C4

C3

C2

B4

B3

B2

A4

A3

A22

3

4

frequen

occasional

rare



D5 E5 C5 B5 A55

extrem

ely rare

ABCD E

seriousmajorsignificantno determined No safety effect

5

D2

D1

E2

E1

C2

C1

B2

B1

A2

A11very frequen

ten

t

AA2

AA1

Risk Classification Scheme for occurrences covering the capability to deliver safe ATM services (European version)

D4

D3

D2

E4

E3

E2

C4

C3

C2

B4

B3

B2

A4

A3

A22

3

4

freque

occasional

rare AA4

AA3

AA2



D5 E5 C5 B5 A55

extrem

ely rare

ABCD E

Partial inabilityto provide safeATM services

Ability to providesafe but degraded

servicesNot determined

No effect on ATM services

AA

AA5

Serious inabilityto provide safeATM services

Total inabilityto provide safeATM services

RAT and the performance IR EC691/2010

• (2012‐2014)• The second European Union‐wide

safety key performance indicator shall be the percentage of application of the severity classification of Risk Analysis Tool as defined in Section 2, point 1(b) below in States where this Regulationbelow in States where this Regulation applies, to allow harmonised reporting of severity assessment of Separation Minima Infringements, Runway Incursions and ATM Specific Technical Events

• The second national/FAB safety KPI shall be the application of the severity classification of the Risk Analysis Toolto allow harmonised reporting of severity assessment of Separation Minima Infringement, Runway



Incursions and ATM Specific TechnicalEvents at all Air Traffic Control Centres and airports with more than 150 000commercial air transport movements per year within the scope of this Regulation (yes/no value).

6

Why RAT?

The RAT is a tool meant for:

• Service Providers – as part of their SMS (The ATS Unit• Service Providers – as part of their SMS (The ATS Unit Manager perspective: the safety performance of his unit )

• Regulators – as part of the Aviation‐wide sector oversight (Representing the airline passenger perspective: how safe is it to travel by air)



Why Risk Analysis?

The Risk Analysis allows setting priorities:

ability

Urgent action

May like/think to do something



severity

repeata

Do nothing

Look Into it

7

Risk Analysis Tool Aim

Ambition was to have a tool that is:

•Simple

•Easy to use

•Efficient/reliable (at least for 80/90% of cases)

Realism is that the tool:

•Will not provide the golden truth



•May not work well for complex incidents

•will still rely on experience and knowledge of experts

Tool principles

RAT (Excel sheet) works as follows:

•Requires the user to answer questions looking as much as

A. SEVERITY

1. Risk of collision ATM ground

ATM airborne

ATM overall

Minimum separation achieved 0 0Separation + 75% minimum 1 1Separation >50%, <=75% minimum 3 3Separation >25%, <=50% minimum 7 7Separation <=25% minimum 10 10

Total separation (a) 0Rate of closure NONE 0 0Rate of closure LOW (<=85knots, <=1000ft/mn) 1 1Rate of closure MEDIUM (>85 and <=205 knots, >1000

d 2000 ft/ ) 2 2

B. REPEATABILITY

3. Systemic issues ATM ground

ATM airborne

ATM overall

Procedures DESIGN 12 12Procedures IMPLEMENTATION 8 8Procedures LACK OF 8 8 0Equipment DESIGN 12 12Equipment IMPLEMENTATION 8 8Equipment LACK OF 8 8 0Human resources management (staff planning, staff assignment, training) DESIGN 12 12q q g

possible at the facts (from investigation report)

•Based on these answers RAT computes proposed Risk classifications for ATM Ground, as well as for ATM (i.e. ground and airborne part)

and <=2000 ft/mn) 2 2Rate of closure HIGH (>205 and <=700 knots, >2000 and <=4000 ft/mn) 4 4Rate of closure VERY HIGH (>700knots, >4000ft/mn) 5 5

Total rate of closure (b) 0

TOTAL (1-ATM) Risk of Collision (a)+(b) 0TOTAL (1-ATM Ground) Risk of Collision (a)+(b) 0

2. Controllability ATM ground

ATM airborne

ATM overall

Conflict detected 0 0Conflict detected late 3 0Conflict NOT detected 5 0 0Plan CORRECT 0 0Plan INADEQUATE 3 0NO plan 5 0 0Execution CORRECT 0 0Execution INADEQUATE 3 5NO execution 5 10 0Loss of separation detected because of STCA 3 0No detection (including by STCA) 5 0 0Recovery CORRECT 0 0

Human resources management IMPLEMENTATION 8 8Human resources management LACK OF 8 8 0

4. Non-Systemic/ Human Involvement issues ATM ground

ATM airborne

ATM overall

Non-Systemic/ Human Involvement issues with Contextual Conditions 12 12Non-Systemic/ Human Involvement issues without Contextual Conditions 8 8 0

TOTAL 3&4a 0

TOTAL 3&4b 0 0

Total (3&4-ATM) = (3&4a)+(3&4b) 0Total (3&4-ATM Ground) = (3&4a) 0

5. Window of OpportunitySituation

Methods

Daily routine

Workload peak

Emergency/ Unusual

situationsnormal 7 5 3degraded mode 6 4 2contingency 3 2 1

Total (5)



Recovery INADEQUATE 5 6NO recovery or the ATM ground actions for recovery have worsened the situation or ATM airborne has worsened the situation 10 15 0TCAS triggered (useful RAs only to be considered) or see and avoid pilot decision (in the absence of TCAS) 10 0NO TCAS RA 0 10 0Pilot(s) followed RA (or, in absence of RA, took other effective action, as a result of see and avoid decision) 0 0Pilot(s) INSUFFICIENTLY followed RA 0 10Pilot(s) INCORRECTLY followed RA (or, in the absence of RA, took other inadequate action) 0 15 0

TOTAL (2-ATM

Ground) 0

TOTAL (2-ATM

Airborne) 0 0

TOTAL SEVERITY :SEVERITY ATM =(1) + (2-ATM) 0

SEVERITY ATM Ground = (1) + (2-ATM Ground) 0

TOTAL REPEATABILITY :ATM =(3-ATM)+(4) 0

ATM Ground = (3-ATM GROUND)+(4) 0

8

•You can savecomments orremarks about yourselections.•It’s limited to 250 characters.



Click to open Causal / Contributing FactorswindowOverall ATM and ATM

Ground ReliabilityIndicators with colourdegraded progress bars

How does RAT work?

(barrier model)

ACCIDENT

(barrier model)

ACCIDENT

Separation(V or H)

Rate of closure(V or H)

CONTROLLABILITYRISK of COLLISION

Systemic & Non systemic issues

Window of opportunity

ATC PILOT

PROVIDENCECONTROL

Separation(V or H)

Rate of closure(V or H)

CONTROLLABILITYRISK of COLLISION

Systemic & Non systemic issues

Window of opportunity

ATC PILOT

PROVIDENCECONTROL



RISK

SEVERITY REPEATABILITY

RISK

SEVERITY REPEATABILITY

9

Where do the figures (used for computation) come from?

A large number of incidents were classified using expert judgement

Numbers used for computation could thenNumbers used for computation could then be worked out (kind of reverse engineering)

ValidationA large number of incidents was used (same and then different from first set)



Comparison and adjustments

Group 1 NOT using RAT

Group 2 using RAT

Next incident(s)Swap groups

Rules for organisation(s) SOP?

Tool limitations Usage biais

Possible inconsistent results/data set

RULES & PROCEDURES

What will RAT be used for? ALL incidents? During and after investigation only after?



What will RAT be used for? ALL incidents? During and after investigation, only after?

Who will be entitled to use the tool? Experience and training requirements

Who will have the final word? Panel chair

Who will use the results and what for? Finding KRAs, trends and follow‐up

How will a panel work? Consensus?

Etc…

10

Rules for organisation(s) SOP – The FAA Example

ApplicabilityThe Risk Analysis Tool is to be applied to radar incidents involving two or more airborne aircraft with a Measure of Loss (MOL) of less than 66 percent. The tool contains separate spreadsheets for wake and non‐wake eventscontains separate spreadsheets for wake and non‐wake events.

In order to prevent the introduction of inadvertent bias to Risk Analysis Tool values, data entry should be completed by a group of experts in possession of as much investigatory and factual information as possible.

Risk Analysis Moderation PanelThe Office of Quality Assurance will determine those events eligible for Risk A l i ll d Ri k A l i E t (RAE ) d ill ll t d t f li bl



Analysis, called Risk Analysis Events (RAEs), and will collect data for reliable analysis.

The panel will consist of three to five experts trained in the use of the tool.These experts will have diverse backgrounds from ATC, human factors, and/or flight operations..

Rules for organisation(s) SOP – The FAA Example

The FAA example (2)Panel Qualification and CurrencyThere are multiple requirements for experts to participate on a Risk Analysis Moderation Panel, including experience, judgment, objectivity, and reliability. In addition, participants must have a clear understanding of the concept of risk as defined in this document and its application in the ATC environment;

Nomination ProcessThe Manager of Quality Assurance will nominate, in writing, candidates for panel participation. Nominations should include a brief summary of the characteristics and qualifications noted above and will be submitted to the Director of Quality Assurance, who will select appropriate candidates from the pool of nominees;;

Member Training and CurrencyTo ensure quality and standardization, panel members must be properly trained and qualified. Once qualified individuals are responsible for maintaining their currency requirements as detailed in this



qualified, individuals are responsible for maintaining their currency requirements as detailed in this section. Panel members must notify the chair if their currency lapses;

In order to receive initial qualification as a Risk Analysis Moderation Panel member, candidates must:• Read and become thoroughly familiar with the most current SOP and otherdocumentation • Attend a one‐day training session conducted by the chair or his/her designee• Observe at least three RAE analyses.

11

FAA FY 2010 non RAEs vs RAEs (Risk Analysis Events)

FY10 loSS and RAEs

Non RAEs

RAEs19%

RAEs

Non RAEs



Number of Events

Non RAEs81%

FY10 Monthly RAE and LoSS count

350

400

FAA FY 2010 non RAEs vs RAEs (Risk Analysis Events) cont’d

100

150

200

250

300

LoSS Events

Number of RAEs



0

50

100

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

12

November 26, 2010

• 552 Total Risk Analysis Events (RAEs) FY10

365 were Operational Errors (OEs)365 were Operational Errors (OEs)

187 were Pilot Deviations (PDs)

• 84 Total RAEs for FY11

51 were Operational Errors (OEs)

33 were Pilot Deviations (PDs)



Leading Causal/Contributing Factors



13

High Risk

Actual results of the 513 events reviewed (Nov 09 – Oct 10), using the ATO SMS risk

i d i k

FAA Risk Assessment Results

Risk Analysis Event (RAE)

1 2 3 4 5

5

Increasing Severity

# OF RAEs

ASSIGNED VALUE

(%OF RAEs

ASSIGNED VALUE)

Medium RiskLow Risk

matrix and risk assessment program jointly developed by FAA & EUROCONTROL

Detailed analyses are triggered by a loss of separation greater than 34% of standard

5

4

3

2

Increasing Likelih

ood

27 123(23 98%)

52(10 14%)

103(20 08%)

118(23%)

4(.78%)

24(4.68%)

18(3.51%)

7(1.36%)

11(2.14%)

1(0.19%)

4(.78%)

4(0.78%)

4(.78%)

3(.58%)

1( 19%)

2(.39%)

1(.19%)



1(0.5%)

separation.

Serious Loss Event = High Risk Matrix Event (Red)

2

1

(5.26%)

3(.58%)

(23.98%) (10.14%)(20.08%)

2(.39%)

1(0.19%)

(.19%)

>=31 18‐30 10‐17 0‐9

not enough

Information

Severityhigh low

Number of Separation Minima Infringements (Risk ATM GND) in RAT‐Risk‐Matrix

Sample of European ANSP Risk Assessment Results

Serious Major Significant No safety effect Not determined

A B C E D‐‐> 4 15 83 124

>=32

1 0 0 0 5SMI

ATM GND

total

24‐31

2 0 1 2 8 226

17‐23

3 1 3 11 23 21

pe

ata

bil

ity

hig

h

for

SM

I, a

s th

ere

is a

lway

s io

n av

aila

ble

for

inve

stig

atio

n as

sess

men

t of r

isk



11‐16

4 1 4 27 34 51

0‐10

5 2 7 43 54 154

Risk to be monitored

Risk to be mitigated

Re

low

No further action

Not

rel

evan

t fen

oug

h in

form

atan

d a

14

FAA Initial results ‐ Issues Identified using RAT

• OJTI performance

• ATC recovery methods• ATC recovery methods

• RNAV procedures and mitigation strategies

• Misapplication of Visual Separation

• Turns to Final



Causal Factors Watch List

• Controller recovery actions to address immediate aircraft conflicts

‐ 41% involved ATC recovery issues

‐ 55% of the recovery weighted events were poor

‐ Out of the most severe RAE 87% involve ATC recovery issues

‐ Most severe score: 5/5



Most severe score: 5/5

15


• Turns to final (same altitude on parallel turn‐ons, poor )vectoring techniques, etc.)

‐ 5% of RAE

• On‐the‐Job Training instructor performance

‐ 10% of RAEs






• PDs and mitigation designs for simultaneous Area Navigation (RNAV) departure applications

f‐ 13% of PD RAEs


• Misapplication of Visual Separation

‐ 9% of RAEs





16

Top Five Causal Contributing Factors

7948

FAA Top 5 ATC Causal Contributing Factors

5150

49



A1-2. Not See A2-1. Forgot to monitor

D3-9. Controller / pilot under training D6-14. Lack of personal perception of risk

A3-2. Incorrect decision / plan

Top 5 Pilot Causal Contributing factors

33 23

FAA Top 5 Pilot Causal Contributing Factors

8636

35



B1-2. Correct pilot readback followed by incorrect action

B1-9. Failed to follow ATC procedure

A2-1. Forgot to monitor

B1-6. CRM issues

B2-2. Pilot failed to / or slow to respond

17

What are the advantages of RAT?

RAT provides for:

•A means to reduce subjectivity

•Harmonisation

•Transparency

•Easier communication/discussion

•Quick process



•Works as “push” to enhance investigation

•Provide hard facts in backing up Recommendation for improvement – see Recovery example



Risk Analysis Tool Process (RAT/RAP) Meetings Seminars and Workshops/Safety... · 1 Risk Analysis...

Documents

Transcript of Risk Analysis Tool Process (RAT/RAP) Meetings Seminars and Workshops/Safety... · 1 Risk Analysis...