Developing a SCMT for European ANSPs · MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS Dr. Barry...
29
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS Dr. Barry KIRWAN, EUROCONTROL, Safety Improvement Research Dr. Kathryn MEARNS, UNIVERSITY OF ABERDEEN, Industrial Psychology Research Centre Dr. Richard J. KENNEDY, BOEING R&T EUROPE, New Programs and Safety 8 th USA / Europe Air Traffic Management Research & Development Seminar (ATM 2009)
Transcript of Developing a SCMT for European ANSPs · MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS Dr. Barry...
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPSDr. Barry KIRWAN, EUROCONTROL, Safety Improvement ResearchDr. Kathryn MEARNS, UNIVERSITY OF ABERDEEN, Industrial Psychology Research CentreDr. Richard J. KENNEDY, BOEING R&T EUROPE, New Programs and Safety
8th USA / Europe Air Traffic Management Research & Development Seminar (ATM 2009)
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 2 / 29
Introduction to Safety Culture in ATM
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 3 / 29
• Safety is not just reliability of ATM/Aircraft equipment and the robustness of the management systems and operational procedures – there are other less tangible considerations.
• Safety Culture is the 'third age of safety' – it has been shown to be a key predictor of safety performance in many industries, including nuclear, chemical, off-shore and rail.
• The concept has been used in relation to ATM although it is not clear how it should be used when applied to ATM – hence the project was born.
Background
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 4 / 29
A Safety Culture Model
“The safety culture of an organisation is the product of individual and group values, attitudes, competencies and patterns of behaviour
that determine commitment to, and the style and proficiency of, an organisation’s health and safety management”
Advisory Committee for Safety on Nuclear Installations (HSC, 1993, p. 23)
What is BelievedWhat is Believed
What is SaidWhat is Said What is DoneWhat is Done
Safety Performance
Safety Performance
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 5 / 29
Safety Cuture Measurement Toolkit
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 6 / 29
Understanding Safety Culture
Understanding Safety Culture
Developing a Safety Culture
Measurement Tool
Developing a Safety Culture
Measurement Tool
2005: Literature Review & Survey of 4 ANSPs
2006: Tool Development & Safety Managers Workshops
Safety CultureEnhancement
Safety CultureEnhancement
Development & Deployment Plan
2009 - 2013: Circa 30 ANSPs surveyed
2007 - 2008: Tool Deployed at 8 ANSPs
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 7 / 29
SAFETY CLIMATE(the ‘mood’)Questionnaires
SAFETY CULTURE(the ‘personality’)Interviews and focusgroups
Focus on Safety Climate & Culture
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 8 / 29
The SCMT Process
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 9 / 29
The 13 Safety Culture Elements
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 10 / 29
SAFETY CULTURE
Commitment
Team work
Just, Reporting & Learning
Involvement
Responsibility
Communications& Trust
The priority given to safety in organisational planning and day to day operations,
both at the management and operational levels Acceptance at the
organisational and individual levels of the responsibility for
safety
Employees’ and Managers’participation in safety
discussions, activities and improvements
Vertical and horizontal communication channels are
efficient, and people have faith in the processes, their
peers and managers
People are willing to report safety occurrences, without fear of being blamed and the Organisation having the will and capability to learn from
safety occurrences
Cooperation and coordination between team members
(within a team, across teams)
Cooperation and coordination between team members
(within a team, across teams)
Safety Culture Themes
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 11 / 29
The Questionnaire
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 12 / 29
The Workshops
• About 5-6 groups of 4-6 people + up to 3 people of assessment team (facilitators/scribes).
• 3 hrs for each session with a break after about 2 hours.
• Assessment team needs to be seen as impartial.
• No management or safety team members in workshops with engineers and ATCOs.
• ‘Inexact’ but pragmatic process to explore issues, ensure understanding, evaluate priority, consider ‘resolutions’.
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 13 / 29
SAFETY CULTURE
Commitment
Teaming
Learning &Reporting
Responsibility
Involvement
Communications& Trust
• Analysis of Safety Culture Questionnaire highlights items / themes where there are ‘issues’for respondents - Identifies what is happening; what people perceive going on around them.
• Workshops explore these issues in more detail & try and identify ‘good practice’ -Determines why it is happening and what might be done to solve the problem.
Outputs from SCMT
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 14 / 29
Deployment of Tool in European ANSPs
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 15 / 29
Deployment Process
Pre‐discussions Pre-launch of SafetyCulture Assessment
Workshops
LaunchSurvey Kickoff
Interviews & Workshops
Analysis & Conclusions
Safety Culture Analysis
Feedback & Way Ahead
Distribution ofQuestionnaires
Familiarisationvisits
Additional DataCollection
Initial Safety Culture Analysis
Interviews
Safety Culture Diagnosis
Initial Analysis
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 16 / 29
Questionnaire: Top 5 Items
1
1,3
3,6
3,6
3,3
2,3
2
2,3
3,6
3,9
96,8
96,7
94,1
92,9
92,8
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Only my manager hasresponsibility for safety
Only the Safety Departmenthas responsibility for safety
Each member of staff hasa responsibility to keep up
with changes toprocedures
If I do something wrong, Ican discuss it with my
colleagues withoutworrying about the
consequences
I have confidence in thepeople that I interact with in
my normal workingsituation
% of respondents
% unfavourable
% neutral
% favourable
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 17 / 29
38.5
35.5
46.4
53
60.3
20.9
26.1
17.9
15.2
12.2
40.5
38.3
35.8
31.8
27.5
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
We have sufficient safetyexperts and support staff in
house
There is a lack of feedback onthe results of safety initiativesthat we have participated in
The other people in theorganization do not understandour job and the safety roles we
fulfil
Feedback from incidentscomes months or years laterwith few recommendations
Even if the system fails, we arestill expected to achieve thetargets that are set for us
% of respondents
% unfavourable % neutral % favourable
Questionnaire: Bottom 5 Items
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 18 / 29
Example Questions: Ops Staff
56.3 16.8 26.9
21.7 23.9 54.3
38.5 15.4 46.2
0% 20% 40% 60% 80% 100%
In ATC everyone know sabout some accident ‘justw aiting to happen’ (C3)
Inmaintenance/engineeringeveryone know s about
‘some accident’ justw aiting to happen’ (E13)
Changes to theorganization and the ATCsystem do not increase
safety-related risks(M20)
Unfavourable Neutral Favourable
Risk Awareness
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 19 / 29
56.3 16.8 26.9
21.7 23.9 54.3
10 30 60
15.4 30.8 53.8
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
In ATC everyone knows aboutsome accident ‘just waiting to
happen’ (C3)
In maintenance/engineeringeveryone knows about ‘some
accident’ just waiting tohappen’ (E13)
I know where controllersdraw the line between safeand unsafe practice (M6)
I know wheremaintenance/engineeringpersonnel draw the line
between safe and unsafepractice (M11)
Unfavourable Neutral Favourable
Example Questions: Managers
Risk Awareness
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 20 / 29
Validation of the Approach
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 21 / 29
• Face Validity:– Does the method / measures look valid? Does it make sense?
• Content Validity:– Does the method or measure represent relevant area of interest? – Feedback from ANSP Managers & comments from ANSP personnel
who have completed the questionnaire indicate that statements make sense to the respondents and cover issues considered to be of importance for good safety performance.
• Construct Validity:– Is the theory behind the method appropriate? Tested statistically by
factor analytic techniques.
• Predictive Validity:– Is desirable but not possible to test at this time.
Validation Criteria
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 22 / 29
Validation of Questionnare: EFA
• Two separate data sets analysed using Exploratory Factor Analysis (EFA) (n=119 & n=310).
• Both data sets revealed a single over-arching ‘Safety Culture’ factor (i.e. most items loaded on this factor).
• Not possible to distinguish separate underlying safety culture ‘themes’ such as ‘Reporting’; ‘Teaming for Safety’; ‘Learning from Incidents’ and ‘Commitment to Safety’.
• Could be issue with small sample size and ratio of items to respondents (i.e. too many items/ too few respondents).
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 23 / 29
Validation of Questionnare: CFA
• Confirmatory Factor Analysis (CFA) determines whether there is a good fit of data to a theoretical model, e.g. safety culture consisting of underlying factors of trust; commitment; communication; etc.
• First stage involved dropping items that were ‘troublesome’ for >15% of respondents, e.g. difficult to understand, N/A.
• 38 items tested across four ANSPs using AMOS 4 with multiple goodness of fit indices (i.e. Chi2, RMSEA, CFI & GFI).
• Series of models run until best fitting model found for ANSP 2 (note this ANSP had largest sample size).
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 24 / 29
Validation of Workshops
• Validity– Good face and content validity
• Consistency– Eurocontrol team use consistent
approach in workshops, applicable toall ANSPs
• Usability– ANSPs need support of Eurocontrol
team to conduct workshops• Insight
– For ANSP 1 another organizationconducted a safety culture surveyfinding similar results
• Auditability– Independent audit of process was
generally positive• Sensitivity
– Approach seems to be sensitive to the context of operations and human factors
• Usefulness– Not evaluated to date
• Resources– Not evaluated to date
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 25 / 29
Is Safety Culture an Issue?
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 26 / 29
A SESAR & NEXTGEN Enabler
• The SCMT approach achieves the objective of understanding the content and processes behind creation of the Safety Culture in ANSPs.
• In a period of change, such as those that will be introduced through SESAR and NEXTGEN, it will be important beforehand to ensure a high-level of Safety Culture across European ANSPs.
• Safety Management Systems (SMS), only when accompanied by strong Safety Culture, will help provide a robust 'safety back-bone' to see ATM safely through this transition to the next generation of ATM.
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 27 / 29
Safety Culture White Paper
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 28 / 29EASS2009 – Nicosia 16-18 March 2009
Chapter 5Safety Culture and Safety
Management Systems
Chapter 1What is Safety Culture?
Chapter 2Safety Culture in context
Chapter 3Changing Safety Culture
SECTION 1Inside Safety Culture
SECTION 2Assessing Safety
Culture
Chapter 1Identifying improvements
Chapter 4: Opening Minds
Chapter 5Learning to learn
Chapter 6Day to day Safety Culture
Chapter 2Walking the talk
Chapter 3Making a difference
SECTION 5FAQ
CASE A
CASE B
CASE C
CASE D
…
Chapter 4Enablers and disablers
Links and references
SECTION 3Improving Safety
Culture
SECTION 4Case Studies of ANSP
experiences
Chapter 2Launching the survey
Chapter 1Getting started
Chapter 3Interviews and
Workshops
Chapter 4Analysis and conclusions
Chapter 5Feedback and way ahead
Chapter 5Re-assessing
Safety Culture Enhancement
http://www.eurocontrol.int/eec/gallery/content/public/videos/Safety_culture_in_ATM.swf
DEVELOPING A SAFETY CULTURE MEASUREMENT TOOLKIT (SCMT) FOR EUROPEAN ANSPS
Napa, 29th June 2009 29 / 29
Thank You for Your Attention
