Post on 16-Apr-2017
Measuring Safety PerformanceAn Analyst’s Perspective
By: Indunil Weerasinghe28th September, 2013
www.linkedin.com/in/indunilmwwww.slideshare.net/walk_the_safety_talk
Measuring Safety PerformanceAn Analyst’s Perspective 2
What are we trying to achieve?
• In Australia:– Civil Aviation Order 82.5 lists AOC
holder’s obligations for HCRPT *:
* HCRPT = High Capacity Regular Public Transport
What are we trying to achieve?
• Internationally:– ICAO framework– Safety Management Manual (SMM) Doc 9859
Measuring Safety PerformanceAn Analyst’s Perspective 3
What are we trying to achieve?
• A key element of an effective SMS is to measure safety performance
• Collect, analyse and report safety data
• Objective: to facilitate safety assurance
Measuring Safety PerformanceAn Analyst’s Perspective 4
Process
•Ensure data quality / integrity
•Extract reactive (lagging) indicators
•Extract proactive (leading) indicators
•Develop predictive analyses
Measuring Safety PerformanceAn Analyst’s Perspective 5
Quality of data
• Data should be:– Valid, complete, consistent, and accurate
• Will always need some clean up, but...• Not feasible to clean it to perfection!
Measuring Safety PerformanceAn Analyst’s Perspective 6
Quality of dataBe pragmatic:
Be mindful of the limitations of your data,but make the most of what you have available!
Measuring Safety PerformanceAn Analyst’s Perspective 7
Three Levels of Analysis
3.Predictive
2. Proactive
1. Reactive
Measuring Safety PerformanceAn Analyst’s Perspective 8
Three Levels of Analysis
• To demonstrate:
– A safety data set was analysed
– Not true operational data (“mock” data used for presentation purposes only)
Measuring Safety PerformanceAn Analyst’s Perspective 9
REACTIVE INDICATORSBasic analyses to support reporting and reactive management
Measuring Safety PerformanceAn Analyst’s Perspective 10
What are “Reactive” Indicators
• Also referred to as “Lagging” Indicators
• Basic measurements that will sum up the recent past
• Can be expressed in terms of simple graphs and charts
• For example....
Measuring Safety PerformanceAn Analyst’s Perspective 11
Examples of Reactive Indicators
Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-130
10
20
30
40
50
60
70
80
90
Incidents (per month) - Whole Fleet
Fleet TotalB777A380E190
?RATE
orQTY
?
Measuring Safety PerformanceAn Analyst’s Perspective 12
Examples of Reactive Indicators• Measuring RATE:
– E.g. Incidents per 1000FH– Communicates a relative measure of risk– Allows for comparison
• E.g. between a/c types, or across timelines
• Measuring QUANTITY:– Communicates an absolute measure of risk– Provides an understanding of the operational impact
• E.g. To determine resources required for incident investigation
Measuring Safety PerformanceAn Analyst’s Perspective 13
Examples of Reactive Indicators
35.3%
29.3%
13.2%
6.4%
5.8%2.4%
2.2% 1.6% 1.0% 2.7%
Incidents by Incident Type
Wildlife
Weather
ATS Deficiency
Laser Light
Aerodrome Related
Other Aircraft Failure to Comply
Forecast Deficiency
Other Operating Environment
Wake Turbulence
Other
Measuring Safety PerformanceAn Analyst’s Perspective 14
Examples of Reactive Indicators
Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Dec-110
10
20
30
40
50
60
Seasonal Variation of Weather Related Incidents
Drops off towards the end of the year (Oct/Nov/Dec)
Measuring Safety PerformanceAn Analyst’s Perspective 15
PROACTIVE INDICATORSBasic statistical analyses to support proactive management
Measuring Safety PerformanceAn Analyst’s Perspective 16
What are “Proactive” Indicators• Classified as a “Leading” Indicator
• Forecasting the near future based on trends
• Aims to flag problem areas before they manifest
• From ICAO’s SMM (Rev 3):– Uses “standard deviation” to set up “alert levels”
Measuring Safety PerformanceAn Analyst’s Perspective 17
Setting the Alert Levels• Calculate monthly mean
• Let’s call this μ
• Calculate the std dev. over the same period
• Let’s call this σ
• Calculations could be done using 12 or 24 month data
Measuring Safety PerformanceAn Analyst’s Perspective 18
When are alerts triggered?
• Reading above (μ + 3σ) in any month
• Reading above (μ + 2σ) in two consecutive months
• Reading above (μ + σ) in three consecutive months
Measuring Safety PerformanceAn Analyst’s Perspective 19
When are alerts triggered?
Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-13 Apr-13 May-13 Jun-1315
17
19
21
23
25
27
29
31
33
35
Incidents Last YE's μ μ + σ μ + 2σ μ + 3σ
Measuring Safety PerformanceAn Analyst’s Perspective 20
PREDICTIVE INDICATORSAdvanced statistical analysis to support predictive management
Measuring Safety PerformanceAn Analyst’s Perspective 21
What are “Predictive” Indicators
• Also classified as a “Leading” Indicator• Less guidance available• Calls for a more creative approach to the
analysis
• This is where understanding your data and knowing how to make the most of it will really pay off!
Measuring Safety PerformanceAn Analyst’s Perspective 22
An example of a Predictive Indicator
• The following statistical test and its application won’t necessarily lie true of all data sets
• It’s just an example to get you thinking about how you could further analyse your data
Measuring Safety PerformanceAn Analyst’s Perspective 23
The Chi square test
• Null hypothesis (H0):– Aircraft type category are
INDEPENDENT variables
• Chi square test on H0 yielded a P score of 0%
• Conclusion - Reject the null:• There is a highly, statistically significant
dependency
Measuring Safety PerformanceAn Analyst’s Perspective 24
Taking away the “Stats Speak”
• A given aircraft type is more vulnerable to incidents of a certain nature
• Opportunity to predict and investigate problem areas regardless of “alert levels”:
– For example:• Risk profile to show high risk areas
Measuring Safety PerformanceAn Analyst’s Perspective 25
What do we mean by a “Risk profile”?
• What if we were to compare...
Actual frequency of a particular type of
incident on a particular aircraft
Theoretical frequency if the incident were to happen independently
of aircraft type
Measuring Safety PerformanceAn Analyst’s Perspective 26
A380 Risk profile “hot spots”
• This tells us “VHF Comms” incidents are 7x more likely on this a/c in my fleet than if a/c and incident types were independent
• Flags up a high risk area, regardless of an alert level being breached
• Warrants a root-cause analysis:– Is there an equipment reliability issue? Pilot training? Procedural issue?
Measuring Safety PerformanceAn Analyst’s Perspective 27
IN CONCLUSION...
Measuring Safety PerformanceAn Analyst’s Perspective 28
Keep in mind....All these examples are simply that... Just fake data designed to convey an
idea.
Don’t apply the conclusion!Don’t even apply the method!
Just apply the thinking!Measuring Safety Performance
An Analyst’s Perspective 29
OpportunitiesChallenges
1. Minimal guidance on “predictive indicators”
As SMS’s mature and enter latter phases of implementation, this is an area that will benefit from increasing industry & academic collaboration.
Potential for significant value addition to the “safety assurance” process
Measuring Safety PerformanceAn Analyst’s Perspective 30
Opportunities
2. Operational safety data tells you whether you are achieving the desired safety outcomes, but...
... do you know if those outcomes are being achieved in the most efficient, least costly manner?
Measuring Safety PerformanceAn Analyst’s Perspective 31
Opportunities A “balanced scorecard” approach to safety
metrics would complement the existing leading/lagging indicators
Better return on your safety investment
Incentivise the desired safety behaviours
Nurture a generative safety attitudes
Cultivate a continuous improvement culture
Measuring Safety PerformanceAn Analyst’s Perspective 32
• Thanks for reading, and remember...
• The aggregate safety of our industry lies in our hands, so...
• ... the more we innovate and share our ideas, the better the outcome we can achieve.
• Please feel free to contact me:– On LinkedIn at www.linkedin.com/in/indunilmw– Or email me at indunil.w@orcaanalytics.com
Measuring Safety PerformanceAn Analyst’s Perspective 33
34
Download these Resources• ICAO’s Safety Management Manual (Doc 9859 Rev 3)
• Robert S Kaplan’s paper from the Harvard Business School on the Balanced Scoredcard
• Testing for Independence using the “Chi Square” Test, from Berkeley University
• Australian Legislative Instrument: Civil Aviation Order 82.5
Measuring Safety PerformanceAn Analyst’s Perspective