The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational...
Transcript of The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational...
![Page 1: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/1.jpg)
Assessing High Reliability Organizational Effectiveness
The High-reliability Organizational Effectiveness Survey (HROES)TM
Anthony Ciavarelli, Ed.D
September 2006
Copyright Human Factors Associates, Inc.2006. All rights reserved.
Prevention First 2006
![Page 2: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/2.jpg)
0
10
20
30
40
50
60
50 65 80 95Angled Decks
Aviation Safety CenterNaval Aviation Maintenance Program
RAG (FRS) Concept InitiatedNATOPS Program
Squadron Safety ProgramSystem Safety Aircraft Design
CRM Aircrew reviewsRisk management(ORM)Safety climate-culture
776 aircraftdestroyed in
1954
Naval Aviation Class ANaval Aviation Class AFlight Mishap RateFlight Mishap Rate
FY50FY50--0505
Fiscal Year
29 aircraftdestroyed in
FY05-all in flightmishaps
96-2005
![Page 3: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/3.jpg)
US Naval Aviation: A high-reliable organization
![Page 4: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/4.jpg)
Naval Aviation Risk Mitigation
• Recruitment and selection• Aeromedical screening• Training standardization and qualifications• Command supervision and risk
management• Aircrew performance reviews• Human Factors reviews (Boards &
Councils)• Crew Resource Management (CRM)• Safety Climate and Culture Assessments
![Page 5: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/5.jpg)
US Naval Aviation: A high-reliable organization
![Page 6: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/6.jpg)
Examples of Organizational Failures
• US Navy F-14 crash in Tennessee • Valu Jet crash into Florida Everglades• Three-mile Island and Chernobyl• Challenger and Columbia• Exxon Valdez Oil Spill• Enron and World Com, others
![Page 7: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/7.jpg)
What Are Common Root Causes?(Organizational)
• Poor risk perception and threat recognition • Lack of leadership commitment to safeguards• Inadequate management oversight and control• Inadequate or unclear risk-decision criteria• Too difficult to report at-risk decisions/behavior• Policy and incentives reward excess risk taking• Culture does not support desired
attitudes/behavior
![Page 8: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/8.jpg)
Organizational Failures: The human factors
• Unhealthy attitudes about safety, risk perception and performance expectations
• Inadequate job qualification standards, training and personnel assignments
• Management tolerance of high risk behaviors• Poor communications and information flow• Non-compliance with rules, or best practices• Poor management of job Stress, fatigue and/or excess production
pressure• Inadequate resources to support tasks assigned• Flawed reward system with little or no reward for safe behavior
Common root Causes(Cultural)
![Page 9: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/9.jpg)
• Failure to instill strong risk values/culture• Pressure to complete to the job, or meet a schedule• Failure to establish or enforce standards• Over-tasking personnel in high-risk jobs• Failure to identify and manage people under stress• Inadequate resources to do job within acceptable risk• Error prone technical equipment, or low reliability• Failure to manage known risks, including high-risk• Permitting High-risk individuals in the workplace
Management Contributions to Failure
![Page 10: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/10.jpg)
What is a high-reliability organization (HRO) ?
An organization that:– Conducts nearly error free operations, with very few
accidents, low incidents, and no disasters– Demonstrates superior performance over a long
period of time– Makes consistently good risk decisions resulting in
high quality, safe and reliable operations
![Page 11: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/11.jpg)
Characteristics of an HRO
• A Culture of trust and shared Values.• Risk mitigating communication processes
that provide opportunities for open discussion and improvement.
• Distributed decision-making, “where the buck stops everywhere.”
(Roberts, 1997)
![Page 12: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/12.jpg)
Organizational Culture
Shared Values (What is important) and Beliefs (How things work) that interact with an organizations structures and control systems to produce Behavioral Norms (The way things work around here).
Adapted from Reason (1997)
![Page 13: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/13.jpg)
Successful Risk Management Culture
• Shared values about what’s risky and what’s not• Common beliefs about how to conduct risk
minimizing operations• Behavioral norms that govern risk-taking,
everyday procedures and precautions• Transmission of values, beliefs and accepted
practices to others in the organization
![Page 14: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/14.jpg)
Measurement of an organization’s ability to effectively and safely conduct their hazardous operations within a healthy risk culture, expressed in terms of leadership engagement and oversight, and with adequate management policies, clear performance standards and practices.
HRO Climate Assessment
![Page 15: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/15.jpg)
1) Process Auditing -- a system of on-going checks and tests designed to monitor hazardous conditions and risks
2) Reward System -- expected social compensation to reinforce desired behavior, and/or disciplinary action to correct undesired
3) Quality Assurance -- policies and procedures for promoting high-quality performance and work products
4) Risk Management – whether or not the organizationcorrectly perceives operational risks and takes corrective action
5) Leadership and Supervision -- policies and procedures and communication processes used to proactively mitigate risks
Five Element HRO Model
Adapted from Libuser and Roberts (1994)
![Page 16: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/16.jpg)
Survey based on HRO Principles
• MEASUREMENT AREAS
• Process auditing
• Reward system
• Quality assurance
• Risk management
• Leadership and supervision
• SAMPLE SURVEY ITEMS
• Our hazard reporting system is likely to identify problems that could lead to a serious accident.
• Senior management supports a climate that promotes safety.
• My [organization] has a reputation for high quality performance.
• I believe that our leaders have a clear picture of operational risks.
• Senior executives at FAA/ATC are genuinely concerned about risk management.
![Page 17: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/17.jpg)
HRO-Climate-Culture Surveys
• Aerospace
https://65.104.119.152/nasa/login.html
https://65.104.119.152/lockmar/login.html
• Aviation
https://65.104.119.152/alaskaair/login.html
https://65.104.119.152/phi/login.html
https://65.104.119.152/bristow/login.html
• Medicine
https://65.104.119.152/totallykids/login.html
![Page 18: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/18.jpg)
Validation Based on 10-years of Navy Research
Reliability Testing – dependability or consistency of measures
Construct Validity- Statistical verification that factors identified support the proposed safety climate model
Predictive Validity- Correlation and statistical verification of relationships between climate measures and safety performance
![Page 19: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/19.jpg)
Average CSA Scores by Military Rank
2
2.5
3
3.5
4
4.5
5
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61
O4 - O6O1 - O3E6 - E9E1 - E5
Rank differences statistically significant (p<.001)
Over-committedShortResources
Survey Items
Military Rank
Trend Analysis: Navy Safety Climate
![Page 20: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/20.jpg)
Example of Navy Validation Data (2001-2004)
MCAS Survey vs Mishaps within 24 Months after Survey
21 1912 13
12 13
93
61
46
33
33
0
10
20
30
40
50
60
70
80
90
100
2.90-3.59 3.59-3.72 3.72-3.82 3.82-4.26
Quartiles based on Survey Average
Mis
haps
Class C
Class B
Class A
94
76
5449
![Page 21: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/21.jpg)
Mishaps within One Year after Taking CSA(Class A: Major Accidents)
Bottom Mid-Low Mid-High Top
Operational Squadrons
All Squadrons
10
6
7
5
9
4
3
2
0
1
2
3
4
5
6
7
8
9
10
Cla
ss A
FM
s w
ithin
One
Yea
r
Quartiles based on Risk Management Average
(Surveys conducted from 01JUN04-31MAY05)
Schimpf and Figlock, 2006)
![Page 22: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/22.jpg)
Mishaps within One Year after Taking CSA(Class A: Major Accidents)
Bottom Mid-Low Mid-High Top
Operational Squadrons
All Squadrons
10
6
7
5
9
4
3
2
0
1
2
3
4
5
6
7
8
9
10
Cla
ss A
FM
s w
ithin
One
Yea
r
Quartiles based on Risk Management Average
(Surveys conducted from 01JUN04-31MAY05)
![Page 23: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/23.jpg)
US Navy verses Hospitals as HROs:Safety climate survey comparison shows much fewer
“problematic” responses for US Navy than Hospitals.
ClimateComm.
mgmt flex
safety issues
0
5
10
15
20
25
30
NavyHospitals
(Gaba, Singer, Sinaiko, Bowen, and Ciavarelli, 2003)
Gaba, Singer, Ciavarelli, et al, 2003
![Page 24: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/24.jpg)
Summary of Validation Tests
• Over 50,000 US Naval Aviation surveys provided a substantial validation sample for the HRO-climate survey.
• Cronbach-alpha reliability 0.90 overall and for each subscale• Factor analysis showed 46% of the survey items are related to
the “organizational (safety) climate” dimension.• Safety outcomes (accidents) are significantly related to safety
climate ratings. Navy squadrons with low safety climate ratings have the highest accident rate, and conversely higher rated units have the lowest accident rates (Ciavarelli, 2005).
• US Navy Safety Climate vs. hospitals tested on same survey items show US Navy Aviation is perceived to out perform hospitals as an operational high-reliability-organization or HRO.
![Page 25: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/25.jpg)
Characteristics of the Survey System
• Founded on a validated HRO climate survey• Anonymity for participants and confidentially for managers• Strategies to maximize participation• Online survey administration• Ease of participation; 18-22 minutes to complete• Online immediate diagnostic results feedback using
normalized scores for equitable comparisons across different organizations and domains
• Trend analysis and intervention strategies
![Page 26: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/26.jpg)
Survey Development Process
• Survey developers meet with unit personnel to discuss the survey system and its potential uses
• Survey developers conduct interviews with a strata of management and operational personnel
• Together developers and unit personnel tailor baseline (paper) survey items to fit a specific domain, and operational environment
• Following approval of “paper” survey, a web-based version is created and a customized web site constructed
• The Web Survey is Beta Tested on a selected (small) sample• Following successful Beta, large-scale data collection and
performance feedback is undertaken on a scheduled roll out.
![Page 27: The High-reliability Organizational Effectiveness Survey (HROES) · 2018-09-24 · Organizational Failures: The human factors • Unhealthy attitudes about safety, risk perception](https://reader033.fdocuments.in/reader033/viewer/2022050310/5f7241fa7b43f162817d706e/html5/thumbnails/27.jpg)
Summary and Conclusions
• Original HRO Climate assessment methodology has been developed, successfully applied in Naval Aviation and US Marine Corp ground forces.
• HRO Climate measures show excellent reliability and have been validated against performance outcomes for use as a risk assessment tool.
• Approach incorporates a complete measurement, analysis and diagnostic display “dashboard” feedback system for managers.
• Comparable online systems have been developed and are now in use by civilian organizations, helicopter transport companies, aerospace, a major airline, and a medical care facility.
• Approach applies to other high-risk industries, such as air traffic control, nuclear facilities, homeland security, oil and gas platform operations, etc.
Demonstration of Web Site: NASAhttps://65.104.119.152/nasa/login.html
(copy url and past to browser)