1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE …
Transcript of 1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE …
Lean Six-Sigma in Aviation Safety:
An implementation guide
for measuring aviation system’s safety performance
Ilias Panagopoulos*, Chris Atkin, Ivan SikoraCity University of London, United Kingdom
1ST INTERNATIONAL CROSS-INDUSTRY
SAFETY CONFERENCE (ICSC)
Amsterdam, 3-4 November 2016
Agenda
Introduction and Motivation
The Conceptual Framework
Implementation Guide
Findings and Conclusions
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Motivation:
What we are measuring….
B777- Malaysia MH370, 227pax +12crewB777-Malaysia MH17, 283pax +15crewA320- Air Asia QZ8501, 162pax + 7 crewA320- Germanwings 4U9525, 150pax + 6crew
Total Damage: 4 airliner aircrafts – 862 fatalities
Calendar Year 2014….the best year ever for airline safety….. (Ascend-Flightglobal)
January’14 December’14
Calendar Year 2015….the safest ever year…. (Aviation Safety Network)
January’15 December’15
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
March 2014 – March 2015
The worst year ever…..
Motivation:
Do we normalise the data…?
- Turbojets or Turboprop? - Passenger or Cargo?- Boeing or Airbus?- Embraer or Bombardier?- Western or Eastern built jet?- During take off or Landing?- Climb, descend or cruise?
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Flight Duty Period
Flight TimeLimitations Runway
Excursion
Top Root CauseLeading Indicator
UnstableApproach
Precision orNon Precision
Root CauseLeading Indicator
UnstableApproachContinuedfor landing
Root CauseLeading Indicator
Crew Fatigue
TimePressure
Root CauseLeading Indicators
Root CauseLeading Indicators
Origin FlightDelay
Major Accidentsor
Serious Incidents
Worst credible outcomeLagging Indicator
PROACTIVEPROACTIVEPROACTIVEPROACTIVE
What do we really need to measure?
SevereWeather
(Wx) Conditions
Unscheduled Maintenance
(Mx)
Root CauseLeading Indicator
Root CauseLeading Indicator
EquipmentFailure
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
FDPFTL
Runway Excursion
Top Root CauseLeading Indicator
UnstableApproach
Root CauseLeading Indicator
UnstableApproachContinuedfor landing
Root CauseLeading Indicator
Crew Fatigue
TimePressure
Root CauseLeading Indicators
Root CauseLeading Indicators
Origin FlightDelay
Major Accidentor
Serious Incident
Worst credible outcomeLagging Indicator
PROACTIVEPROACTIVEPROACTIVEPROACTIVE
Assuming, we are measuring Unstable Approaches
What is our Acceptable Level of Safety (ALoS)?
SevereWeather
Conditions
Unscheduled Mx
Root CauseLeading Indicator
Root CauseLeading Indicator
EquipmentFailure
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
2 out of 10..?5 out of 30..?8 out of 50..?10 out of 1000FH..?
FDPFTL
Runway Excursion
Top Root CauseLeading Indicator
UnstableApproach
Root CauseLeading Indicator
UnstableApproachContinuedfor landing
Root CauseLeading Indicator
Crew Fatigue
TimePressure
Root CauseLeading Indicators
Root CauseLeading Indicators
Origin FlightDelay
Major Accidentor
Serious Incident
Worst credible outcomeLagging Indicator
PROACTIVEPROACTIVEPROACTIVEPROACTIVE
Assuming, we have defined ALoS
What is our Acceptable practical drift?
SevereWeather
Conditions
Unscheduled Mx
Root CauseLeading Indicator
Root CauseLeading Indicator
EquipmentFailure
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Safety Target: No more than 10 out of 1000FH
+/-10%..?+20 %... ?+2 or +3..?By when..?
Air Asia A320, Flight QZ8501, 28 December 2014
ALoS???
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The plane's rudder control system malfunctioned 4 times during the flight - a fault that occurred 23
times in the preceding year
Malaysia B777, Flight MH370, 8 March 2014
ALoS???
IAW Cargo manifest:2.5 tonnes of Lithium-ion batteries in the cargo hold of a pax a/c…!!!
‘Aviation safety is the state in which risks associated with aviation activities are reduced and controlled to an acceptable level’
ICAO Annex 19
Since 2009, in an effort to achieve an Acceptable Level of Safety (ALoS) performance
in the Air Transport industry, regulatory authorities mandate air operators
to implement a Safety Management System (SMS).
In sequence, SMS mandates the continuous measuring and monitoring
of safety performance
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
According to ICAO each service provider shall, as a minimum:
1) Establish an SMS
2) Identify safety hazards.
3) Ensure remedial action to maintain agreed safety performance.
4) Provide continuing monitoring and regular assessment of safety performance.
5) Aim at a continuous improvement of the overall performance of the SMS
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Safety Performance (SP) is the State or a service provider’s safety achievement as defined by its SP Targets (SPTs) and SP Indicators (SPIs).
Nevertheless, in Air transport Industry the process for measuring System’s Safety Performance has not yet been introduced or standarised.
Research twofold Problem
How could aviation organisations have a proactive and performance-based approach to safety that focuses on desired,
measurable outcomes and on the management of operational risks?
How could an aviation organisation measure its overall safety effectiveness against performance goals and examine
safety performance variability from core organisational objectives?
Key Research Questions
What methodology could proactively measure system safety performance and improve the safety performancemeasurement process?Could a conceptual framework assist the continuous improvement of the safety performance measuring process?
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
To introduce a conceptual framework for measuring system’s safety performance and performance variability from core organizational objectives.
In addition, the study aims to provide an implementation guide on how air operators coulddesign and develop a proactive, performance-based methodology for measuring AcceptableLevels of Safety Performance (ALoSP) at sigma (σ) level, a statistical measurement unit.
Research Aim
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Agenda
Introduction and Motivation
The Conceptual Framework
Implementation Guide
Findings and Conclusions
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The conceptual framework
The Safety-Performance Indicator Lean Sigma (Safety-PILS) model embedded within the
Define-Measure- Analyse-Improve-Control (DMAIC) continuous improvement process.
The Safety-PILS model
provides a holistic view on how organisations could set:
-leading indicators and monitor metrics on the top of identified root-causes
-lagging indicators and feedback metrics on the top of safety outcomes
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The Safety-PILS model
aims to control and maintain safety performance within agreed Upper or Lower Specification Limits and to develop an objective methodology that will proactively investigate and measure system performance variability within ±1.5 sigma (σ) from an ALoS Performance target.
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Through DMAIC process
the operator could apply Lean-Six Sigma (L6S) methodology for measuring both the performance of each established indicator and system safety performance variability
at sigma level from core safety objectives.
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Agenda
Introduction and Motivation
The Conceptual Framework
Implementation Guide
Findings and Conclusions
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The implementation guide is divided in two phases,
Phase-I and Phase-II.
Phase-I is mainly the utilisation of the Safety-PILS model
Phase-II the practical implementation of the DMAIC
process.
Both Phase I and Phase II are forming the conceptual
framework.
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The conceptual framework
for measuring system’s safety performance
Phase-I.
Safety-PILS model utilisation
1. Design Safety-PILS model for VOB
- Define the driven KPI for the VOB
- Set the VOB Targets and the LSL-USL based on industry standards
- Set SPIs on the VOB- Critical to Safety (CTS) characteristics
- Set metrics on each VOB SPI and the associated LSL-USL
2. Correlation and Multiple Regression Analysis (or Pareto Analysis)
- Identify correlation between cause/effect
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Phase-II. : Apply Six Sigma-DMAIC methodology
3. Data Collection Planning (DCP) for Hypotheses tests
- Hypothesis Testing - Data normalization
4. Control Chart selection - road map
- Control Chart selection for each VOB SPI and Metric
- Identify special causes: If none the process is In-Control
5. Measurement System Analysis (MSA)
- Where does the variation of data comes from?
- Is the process Accurate and Precise?
6. Process Capability
- Is the process capable (i.e. efficient)?
- At what sigma level?
7. Analyse the data
- Identify root cause and attractive areas for improvement
- Identify best and feasible solutions
8. Pilot solutions
- Demonstrate that piloted solution provides a Return of Investment (ROI)
9. Define Control Plan and Roll-out improvement
- Monitor the Control Plan to sustain the change
10. Measure total system safety performance1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)
Amsterdam, 3-4 November 2016
The conceptual framework
for measuring system’s safety performance
Phase-I.
Safety-PILS model utilisation
1. Design Safety-PILS model for VOB
- Define the driven KPI for the VOB
- Set the VOB Targets and the LSL-USL based on industry standards
- Set SPIs on the VOB- Critical to Safety (CTS) characteristics
- Set metrics on each VOB SPI and the associated LSL-USL
2. Correlation and Multiple Regression Analysis (or Pareto Analysis)
- Identify correlation between cause/effect
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The Table shows two indicative indicators, with their associated metrics. SPI_02 and SPI_06
have been selected as a research sample for further explaining and validating the
conceptual framework implementation step guide.
SPI_02: Runway excursions (RE) SPI_06: Loss of Control (LOC)
Metric 02.1: Deep Landing events Metric 06.1: Stick-shake and alpha floor events
Metric 02.2: Unstable-De-stabilized approaches (all) Metric 06.2: Take off Configuration warnings
Metric 02.3: Unstable-De-stabilized approaches (all) continued for
landing
Metric 06.3: Low speed during cruise events
Metric 02.4: High speed touchdown events Metric 06.4: Low speed during approach events
Metric 02.5: High speed rejected take-off events Metric 06.5: Percentage of pilot’s readiness rate for proficiency
Metric 02.6: Take-off landing events involving loss of aircraft directional
control caused by contaminated runway surface
Metric 06.6: Pilot’s utilisation effectiveness
Metric 02.7: Runway and Overrun events due to runway contamination Metric 06.7: Percentage of pilots received upset recovery training
Metric 02.8: Proportion of licensed aerodromes using new reporting
criteria for runway surface condition
Metric 06.8: Percentage of qualified and current pilot’s availability
rate
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Safety-PILS model utilization for the design of the performance indicators SPI_02 & SPI_06
and its associated metrics. For the purpose of this study,
Metric 02.2: Unstable-De-stabilized approaches (all)
Metric 06.2: Take-off configuration warnings events
have been selected as an indicative sample for validating the conceptual framework implementation step guide
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The research study examined the correlationexist among VOB indicators and metrics andrevealed a moderate to strong correlation,since all examined R values are ranging from0.6- 0.8.
In addition, the multiple regression analysisof the study examined the VOB indicatorsand metrics and revealed with 95%confidence that the selected VOB SPIs andmetrics are important or very importantfactors to the VOB process, since allexamined R-Squared values are ranging from44.74 - 83.38%.
Also, the study revealed that Metric 02.2 (i.e.Unstable approaches), is the one thataccounts the most for the variation in theSAC VOB process output.
C8 = 1.0536 - 0.279 X1 + 0.0768 X1^2
Step Change Step P Final P
1
Add X1^2
Add X1
0.021
0.263
0.021
0.263
1007550250
R-Squared(adjusted) %
1.1876484560
0.5938242280
151050
Increase in R-Squared %
43210
1.00
0.75
0.50
0.5938242280
C8
1.1876484560
0.5938242280
100500
R-Squared %
X1: Unstable App X2: T/O Conf. Warnings
X1: 0.5938242280 X2: 1.1876484560
Final Model Equation
Model Building SequenceDisplays the order in which terms were added or removed.
Incremental Impact of X VariablesLong bars represent Xs that contribute the most new
information to the model.
Fitted Line Plot for 0.5938242280
Shows the relationship between C8 and 0.5938242280.
Each X Regressed on All Other Terms
Gray bars represent Xs that do not help explain
additional variation in Y.
A gray bar represents an X variable not in the model.
Multiple Regression forVOBModel Building Report
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Phase-II.
Apply Six Sigma-DMAIC methodology
3. Data Collection Planning (DCP) for Hypotheses tests
- Hypothesis Testing – Data are normalised
4. Control Chart selection - road map
- Control Chart selection for each VOB SPI and Metric
- Identify special causes: If none the process is In-Control
5. Measurement System Analysis (MSA)
- Where does the variation of data comes from?
- Is the process Accurate and Precise?
6. Process Capability
- Is the process capable (i.e. efficient)?
- At what sigma level?
7. Analyse the data
- Identify root cause and attractive areas for improvement
- Identify best and feasible solutions
8. Pilot solutions
- Demonstrate that piloted solution provides a Return of Investment (ROI)
9. Define Control Plan and Roll-out improvement
- Monitor the Control Plan to sustain the change
10. Measure total system safety performance
- Voice of the safety Process (VOP)= VOB
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Data Collection Planning (DCP)
for hypotheses test
Decide on:
• what type of data is most appropriate to
collect for measuring the VOB SPI and
metrics,
• what resolution is needed,
• what statistical tool should be used to
interpret the data and
• what the sample size and frequency
should be.1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)
Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Hypotheses test: Normalise the data
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Hypotheses test
43210-1
4
3
2
1
0
Mean 1.568
StDev 1.185
N 23
0.5938242280285035
Freq
uen
cy
Normal
Histogram of Unstabilised App
3210-1
5
4
3
2
1
0
Mean 1.220
StDev 0.9272
N 23
1.187648456057007
Freq
uen
cy
Normal
Histogram of T/O Configuartion Warnings
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
2321191715131197531
4
2
0Su
bg
rou
p M
ean
_X=1.394
UCL=3.559
LCL=-0.771
2321191715131197531
2
1
0MR
of
Su
bg
rou
p M
ean
__MR=0.814
UCL=2.660
LCL=0
2321191715131197531
4
2
0
Sample
Sam
ple
Ran
ge
_R=1.262
UCL=4.124
LCL=0
I-MR-R/S (Between/Within) Chart of Unstable App and T/O Conf, warnings
Control Chart selection: Process is in-control
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Process Capability: Is the process capable?
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Data Analysis
• Cp> Cpk meaning that the potential process for all metrics was centered.
• Cpk is less than 1, meaning that not only a special cause but also a common cause
of variation was going to produce unacceptable variation (i.e. defects).
• Cp=0.17 means that only 17% of the process fit within USL/LSL and Cpk=-0.08
means the process was 80% over one specification limit.
• The control charts revealed that there was no special cause of variation,
meaning that this process was in-control.
• However, neither the actual process nor the potential was capable.
• At this point the operator needs to apply solutions or to take mitigation measures
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Process NOT in control
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Normality Test
Results Pass
P-value 0.448
(Anderson-Darling)
1.2
0.9
0.6
Mea
n
121110987654321
0.30
0.15
0.00
Ran
geXbar-R Chart
Confirm that the process is stable.
Normality Plot
The points should be close to the line.
Capability Analysis for VOB before (C8)Diagnostic Report
Before implementing Solutions• The actual process is centered
• The process is performing at 0.98+1.5 sigma level (i.e. z actual
value), which is equal to 2.75 sigma performance.
• After implementing solutions, the process will perform at
2.21+1.5 sigma level (i.e. z potential value), which is equal to
3.71 sigma performance.
• The data are following normal distribution pattern since the P
value=0.441which is greater than the threshold value of 0.05.
• The defect rate is 16.43% which estimates the percentage of parts
from the process that are outside the specification limits.
• The DPMO is 164248, meaning that the operator currently
experiences 164248 defects per 1 million flying hours or 164 defects
per 1000 flying hours related to unstable approaches and take-off
configuration warning events.
• The actual process mean which is 0.90783 does not differ
significantly from the target which is 0.94.1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)
Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
• It appears that the actual capability is
improving, with 7.97% of the process being
out of the specification limits with 79711
DPMO.
• In addition, the process is centered still, since
Cp>Cpk.
• This report verifies that data are continuing to
follow a normal distribution since P
value=0.564 which is greater than the
threshold value of 0.05.
• Besides, the Xbar-R chart shows none special
causes for the existing variation.
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
After implementing Solutions
Agenda
Introduction and Motivation
The Conceptual Framework
Implementation Guide
Findings and Conclusions
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Before / After Capability Comparison
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Summary Results
The process performance report shows the improvements achieved
`AFTER the implementation of solutions as follows:
• The percentage out of the specification limits has been significantly reduced by 51%, meaning from
16.43% before to 7.97%.
• The STDV (i.e. variation) was significantly reduced by 7%.
• The actual process performance has been increased by 0.52%
• The sigma level has been increased by 43%,
• The DPMO have been reduced by -84577.
• The actual sigma level performance is 1.41+1.5 sigma which equals to 2.91 sigma level performance.
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Measuring total system safety performance
Voice of the Safety Process (VOP) = VOB = Total system’s safety performance
At the end of Sep 2016 the VOB (i.e.
occurrences rate) had achieved its target 0.06,
meaning that the total percent defective is
0.068% and the percentage yield or
acceptance rate is 99.32%.
The results of 0.046% or 99.954% indicate that
the total system’s safety effectiveness was
approaching 4 sigma performance with the
potential to have 6210 defects per million
opportunities (i.e. occurrences per flying
hours).
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
Recommendations
1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016
The paper recommends the application of the conceptual framework to:
- different settings,
- different sample,
- different type of SPIs/metrics, and
- qualitative validate the results by interviewing Subject Matter Experts (SMEs).
As further research the study recommends:
- the application of Genetics Algorithms and Simulation to the implementation
guide. In this case, a metaheuristic procedure could sample a set of solutions
and select or generate an algorithm that provides a sufficiently good solution to
this safety performance optimization problem.
Lean Six-Sigma in Aviation Safety:
An implementation guide
for measuring aviation system’s safety performance
Ilias Panagopoulos*, Chris Atkin, Ivan SikoraCity University of London, United Kingdom
1ST INTERNATIONAL CROSS-INDUSTRY
SAFETY CONFERENCE (ICSC)
Amsterdam, 3-4 November 2016
Thank you for your attention!