Root Cause Failure Analysis: Fact or Fiction? Part 2 · Root Cause Failure Analysis: Fact or...

Root Cause Failure Analysis: Fact or Fiction?

Part 2

by Namik Kosaric P.Eng. Principle Consultant, Norcan Reliability Engineering

Norcan Reliability Engineering Proprietary – Do Not Copy www.norcanreliabilityengineering.com of 15

Challenge of Incident Investigation

The RCFA Inductive Process, evaluating the risk due to the degraded PSM elements, is a systematic method in incident

investigation to analyze the integrity of current PSM system as defined as policies and procedures. PSM system should

ensure that changes do not result in operations outside of established safety parameters. This need to become essential

component of a plant’s process safety system as degradation and non compliance can occurs daily in a hydrocarbon

processing plant.

Hydrocarbon Processing Plant Complexity The Incident Investigation Team challenge is in obtaining knowledge and understanding of how work and decision making is carried out within current organization, of the current PSM management system and PSM management system effectiveness. Investigation team challenge is to assess:

� Operation Changes: The operating plant environment is changed on the daily basis. Although highly automated

hydrocarbon processing plant are subject to frequent operation parameter changes as well as equipment failures. There

could be thousand of alarms that go on and are left unattended in one day of operation. In addition, hydrocarbon

processing plants are becoming more complex and tightly coupled between process units and operating plants. And

interconnections between plants and functions are frequently not automated.

� Equipment Modification: Hydrocarbon processing plants are exposed to flux of changes implemented by the personnel

involved in redesign, operation and maintenance. Large hydrocarbon processing complex can issue thousands upon

thousands maintenance work orders per year as well as to be involved in large number of abnormal operating scenarios!

Frequently employees are overwhelmed by magnitude of changes and issues.

� Technological Advances: Process changes, improvements, as well as technological advances result in an increase in competence requirements. As the equipment complexity increases, understanding of the equipment implications relative to maintenance and operating change can becomes critical element for safe operation. A characteristic feature of the complex hydrocarbon processing plan is that their complexity often exceeds employee grasp of how facilities are designed and how should they be operated and maintained.

� Redundancy: Current high prices of crude oil have put pressure on production and processing capability of existing

facilities. In order to meet targets availability, equipment & system redundancy is implemented. A high level of redundancy

is easily justified for increasing reliability, and possibly safety. Reliance on redundancy may lead to decreased emphasis

on PM, monitoring, operating envelopes, as well as other safety engineering techniques. In practice, redundancy may

‘cover up’, or mute, operating, maintenance and design errors and prevent them from becoming visible until something

catastrophic occurs.

The Investigating Team challenge is to identify causes, ineffective management system, that drive human performance by

comparing to the PSM system benchmark, i.e. plants with pacesetting HSE and economic performance.

Example: Pump and Electrical Motor Failure Application of the RCFA Inductive Process considering risk of the degraded PSM elements will be illustrated by failure analysis of Starhill Platform Crude Transfer Pump. The example is illustrative and has no resemblance to any actual installation, facility, country or company. The pump was installed in 1988 and was operating with expected reliability till year 2000. Six pump and motor failures have occurred in the last four years, four have required replacement with a factory new motor at a cost of US$ 150,000 each. Failures are characterized by failed bearings, a bent rotor shaft and damage to rotor and stator from heavy rubbing contact.


Part 2



Figure 13: Example Investigation: Pump and Electric Motor Failure

Sequence of Events: Pump and Electrical Motor Failure

The Investigation Team has developed the Sequence of Events for Pump Failures at Starhill Platform, describing timing, who is

involved, details of the sequence, outline / definition of performance less than expected for that sequence, agent involved in this

change (PSM procedure or inadequate facilities), and consequence of performance less than expected. There are two distinctive

periods that can be observed from the sequence of events, expected operation from 1988 till 2000, and high failure rates after year

2000.


Part 2



Figure 14: Sample Sequence of Events

The Investigation Team using the Sequence of Event, Fig. 14, has identified the change, “performance less than expected” and procedure / standards / practices that used control this step. Focus of investigation is a period of high failure rates. Each Event in the Sequence, Fig.14, are evaluated for Active, Condition and Supervision and PSM System failure.

The Investigation Team Analysis of the period after year 2000 has probed deep enough to uncover issues at Starhill platform. Issues that have surfaced have been created due to inadequate design, fabrication, construction as well as poor operation or maintenance. Everything happens for reason and those need to be fully understood to prevent reoccurrence Deviation Statement – Physical Failure Mechanism: Pump and Electrical Motor Failure The Investigation Team has identified considering Physical Failure Mechanism that change in operating condition involving “Water, Sand and Slugs Carry Over” was a significant factor that warrants further investigation of this event. The following is the outline of the Investigation Team consideration of Physical Failure Mechanism:


Part 2



Figure 15: Outline of Physical Failure Mechanism


Part 2



Deviation Statement – Steps in the Sequence of Events: Pump and Electrical Motor Failure The Investigation Team has defined in each Event in the Sequence of Events: Details of Event, Change (Performance Less than Expected), Agent of Change and Effect of Change.

Figure 16: Assessment of Step in the Sequence of Events

Scope of Investigation – Active, Precondition and Supervision, Degraded PSM system: Pump and Electrical Motor Failure The Investigation Team has analyzed the “Operating Condition involving Water, Sand and Slugs Carry Over” by considering Active, Precondition and degradation of PSM elements.

Investigation is performed by application of qualitative techniques defining assessment dimension with rating for completeness and effectiveness considering PSM system degradation. RCFA Inductive Process, considering PSM degradation, is a technique backed up by web based software with quite extensive questionnaire involving assessment of completeness and effectiveness of each PSM dimension.

The following, Fig.17, is outline of the RCFA Inductive Process management system investigation Menu:


Part 2



Figure 17: Scope (Menu) for the Event Assessment

The Investigation Team directed the investigation for each a sentinel event identified for analysis. The members of the Investigation Team were trained in the RCFA Inductive Process techniques and its goals, as the tendency to revert to personal biases is strong.

Multiple investigators and management interference can allow triangulation or corroboration of major findings and increase the validity of the final results.


Part 2



PSM Element Completeness: Pump & Electrical Motor Failure

Investigation Team utilizes criteria’s to assess the completeness and effectiveness of each element of PSM. Those criteria’s require assignment of ranking in values from 1 to 4.

PSM Elements Effectiveness: Pump & Electrical Motor Failure

The following, Fig. 18, is the outline of effectiveness scoring:

Criteria for Assessing PSM Elements: Effectiveness

Applicable

PSM ELEMENTS EFFECTIVNESS

ASSESSMENT Yes Current State Maximum Score

Leaders Focus on Setting PSM Direction and Strategyx 1 4

PSM Element implementation is per Individual ability based Competence Assessment

x 1 4

Each Work Group has PSM Element Implementation defined by Competence Assessment

x 1 4

Each PSM Element Implementation Group has a

Designated Leaderx 1 4

Effective PSM Goals used by Work Groupsx 1 4

Required PSM Competencies are Defined for all Positionsx 1 4

PSM Roles for Operations and Mtce based on Operator

leadershipx 1 4

All PSM Implementation Groups are Organized Similarly

at each Sitex 1 4

Process Shift Teams are Highly Skilled on PSM elementsx 1 4

Effective Coordination between Shift Teams and Day

Work relative to PSM elements

x 1 4

Single Accountability for PSM Performance Measures

(KPI)x 1 4

Figure 18: Criteria for Effectiveness Assessment: Pump & Electric Motor Failures

PSM Elements Degradation: Pump & Electrical Motor Failure

The Investigation Team used the software with build in methodology to assess PSM elements completeness and effectiveness as related to each step of the Sequence of Events. The Investigation Team assessment of PSM Elements as related to the Pump Failures at Starhill Platform, Sequence of Events, Event 02, Period of Frequent Failures, is:

� PSM Mechanical Integrity Assessment: Pump and Electrical Motor Failure The Investigation Team following RCFA Inductive Process questionnaire has examines the content of the knowledge base that should be available to operator and its lack of completeness and effectiveness. The Investigation Team assessment of Sequence No.2 relative to PSM Mechanical Integrity is partially shown below:


Part 2



Evaluate

“Mechanical

Integrity”

Figure 19: PSM Mechanical Integrity Assessment


Part 2



As per assessment of the Investigation Team, there is no “Defect (Failures) Control” at Starhill platform. The operator and mechanics were not provided baseline mechanical integrity data and data on past repair was not available, Fig. 19 The Investigation Team has provided very low rating on operator’s awareness of equipment date base, limitations, PM program and equipment maintenance data. The equipment maintenance data base can include Equipment Maintenance and Reliability Strategy, Reliability Targets, Operating Limits, Material of Construction, specification sheets, time consumed, impact, and costing. Schedule, Annual Preventive Maintenance calendar need to be available and displayed for operators, with work instructions. Data should be made available of values derived using calibrated instruments as compared to Design Specification. Agreements on deviations to schedules and any resultant impact should be documented and provided to operators. The Investigation Team has provided a low rating for Operator involvement in PM and Operational database, understanding of Mean Time Between Failure information for increasing On-time service hours, knowledge of base for Predictive Maintenance, Reliability Engineering data for equipment and components, Plant Life Extension studies , Remnant Life Analysis and Sparing philosophy. The Investigation Team has provided a low rating for testing and inspection programme, Engineered Controls, Vents, Drains, Flare Lines, Elbows, Bends, Turbulent Spots, Spring hangers, Pipe supports, Foundation Settlements, Stack Top Segments, Critical Isolation Valves and Non-Return Valves. The Investigation Team has provided a low rating for inspection techniques used to ensure integrity, reference to appropriate Inspection Standards, Individual Competency, Instrument and Result interpretation.

� PSM Training Assessment: Pump & Electrical Motor Failure The Investigation Team has assessed the PSM Training, as per questionnaire on Fig. 20. Assessment of training is essential to understanding of cases leading to failure in Sequence No.2


Part 2



.

Evaluate

“Training”

Figure 20: PSM-Training


Part 2



The Investigation Team has provided a low rating for Training, no clear Goals and Objectives, not based on Task analysis,

Skills Gap analysis, personnel evaluation criteria, with a structured calendar, participant profile, venue selection for ambience,

Trainer choice, duration prescribed, subject topics relevant to the participants. Employees at Starhill platform were overwhelmed by

magnitude of changes and issues. The constraint at Starhill Platform is employee understanding of processing technology and the

systems in which facilities design, operation and maintenance is embedded.

The Investigation Team has provided a low rating for Levels of Training as published both for budgetary purposes as well as progress evaluation. This should be treated as an on-going activity preferably with a specified budget and an absentee list. A characteristic feature of the Starhill Platform is that the complexity exceeds employee grasp of how facilities are designed and how should they be operated and maintained. In complex, automated systems, such as Starhill Platform, management system accidents are predominant.

Completes and Effectiveness of PSM Elements: Pump & Electrical Motor Failure The Investigation Team has provided completeness and effectiveness rating for all PSM elements relative to failure analysis of Starhill Platform Crude Transfer Pump.

The Investigation Team by using the RCFA Inductive Process proprietary software and methodology has summarize, Fig. 21, a degradation assessment of all PSM element as related to Event 02 in the Sequence of Events.

The Investigation Team has provided rating for the degraded PSM element to prioritize elements where a departure from standard procedures or specifications in PSM results in non-conforming processes or where there have been unusual or unexplained events which have the potential to impact on production, system integrity or personal safety.

The Investigation Team rating for PSM degradation is summarized below:

“PSM

Evaluation

Summary

Results”

Figure 21: PSM Evaluation Summary Score

Risk Assessment due to Degraded PSM System: Pump & Electrical Motor Failure The Investigation Team rating for PSM degradation relative to Events in the subject Incident Investigation, as summarized in Fig.21, are evaluated for the risk due to degraded PSM system.


Part 2



The Investigation Team assessed the risk due to degraded PMS, Fig. 22, by using the PSM system completeness and effectiveness variance, shown in Fig.21, of analyzed steps in Sequence of Events. The Investigation Team risk rating is:

� Risk associated with noncompliance to PSM element for Event 02 of the Sequence of Events is CRITICAL, High Variance and High Consequence,

� Risk associated with that hazard need to be reduced minimum variance to PSM � Investigation Team to determine appropriate ways to eliminate or control the hazard, variance to PSM.

The investigation results need to ascertain that deviation to PSM is a common cause of accidents, by managing PSM system we are managing potential incidents.

1

2

Figure 22: RCFA Inductive Process Risk Metrics


Part 2



Defining Improvements to Reduce Risk: Pump & Electrical Motor Failure

The Investigation Team RCFA has defined improvements, Fig. 23, and prioritizes risks associated with PSM degradation and followed by application of resources to minimize, monitor, and control the probability and/or impact of failures / incidents or to maximize the realization of opportunities.

Figure 23: RCFA Inductive Process Root Cause and Improvements Action


Part 2



The Investigation Team has provided recommendations dealing with fundamentals of PSM in employee involvement, hazard analysis, operating procedures, training and mechanical integrity. The recommendations define improvements and manage risk due to PSM degradation include transferring the risk to another party, avoiding the risk, reducing the negative effect of the risk, and accepting some or all of the consequences of a particular risk.

For the Investigation Team recommendations prioritization is used, whereby the risks with the greatest loss and the greatest probability of occurring are handled first, and risks with lower probability of occurrence and lower loss are handled in descending order.

Implementing Improvements: Pump & Electrical Motor Failure Investigation Team recommendations incorporate fundamental culture change to avoid resistance:

� Fundamental revisit to PSM implementation to offset manager’s resistance to change existing structures. � Fundamental revisit to PSM implementation to offset labor force resistance due to fears of additional work, job

security, layoffs!

Way Forward: Pump & Electrical Motor Failure The Investigation Team will uncover what they are looking for. If there is a common belief that accidents take place due to faults in

the manufacturing management system, they will probably focus on deteriorated PSM system.

However, if the shared belief is that fast consensus is preferable, that brainstorming is the way to go.

Safety and effectiveness vary significantly in the industry. Similar plants can spend third in maintenance cost to achieve pacesetter

safety, availability, reliability and product quality.

Hydrocarbon Processing Industry needs to have an appreciation of a benchmark, how the stellar performance is accomplished by the best in industry. For hydrocarbon processing plants with poor safety and operating effectiveness, plant output does not correlate with resources utilized: the resources are consumed, but the resource consumption stubbornly fails behind the potential accomplished by pacesetters. Poor performing hydrocarbon processing plants feature operating and maintenance teams not equipped to solve safety and reliability problems, poor availability, operating equipment beyond design limitations, poor repair, safety and maintenance effectiveness, ineffective contractor policy, large and ineffective spare material inventories and etc. Way forward for hydrocarbon processing industry could be to promote a benchmark model for PSM effectiveness and use this as a guide for PSM assessment and incident investigation.

List of References:

[1] FMRIMS@Root ™, http://www.norcanreliabilityengineering.com

[2] Swain, A.D. & Guttmann, H.E., Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications. 1983, NUREG/CR-1278, USNRC.

[3] Humphreys, P. (1995). Human Reliability Assessor’s Guide. Human Factors in Reliability Group.

[4] "The Human Factors Analysis and Classification System (HFACS)," Approach, July - August 2004. Accessed July 12, 2007.

[5] Reason, J.[1990] Human Error. Cambridge University Press

[6] Standards Association of Australia (1999). Risk management. North Sydney, N.S.W: Standards Association of Australia. ISBN 0-7337-2647-X.

[7] United States Environmental Protection Agency (April 2004). General Risk Management Program Guidance. United State Environmental Protection Agency. http://www.epa.gov

[8] OSHA PSM 1910.119; Process safety management of highly hazardous chemicals http://www.osha.gov


Part 2



Namik Kosaric is a Canadian Professional Engineer with

experience with Norcan Reliability Engineering (current),

PETRONAS, Bahrain Petroleum Company and ESSO

Petroleum Canada in reliability improvements and

maintenance cost reduction, mechanical design, project

engineering and technical support of Oil Refineries and

Oil Production Facilities.

For the last 8 years in PETRONAS Namik Kosaric was

responsible for providing technical and knowledge

leadership in development, coordination and

implementation of plant reliability and integrity

improvements and program to PETRONAS OPU’s to

improve and support the overall Petroliam Nasional

Berhad objectives.

In BAPCO, Namik Kosaric, pioneered and implemented a

root cause failure analysis of lost profit opportunities and

chronic failures using a multi-disciplinary teams to

improve plant reliability, availability, safety and to

ultimately reduce operating costs. Significant cost savings

were achieved as a result of over 200 completed

investigations.

For 23 years in ESSO Petroleum Canada, Namik Kosaric

has made significant contribution worldwide in reliability

improvements, design, projects and maintenance cost

reduction in upstream and downstream facilities.

Root Cause Failure Analysis: Fact or Fiction? Part 2 · Root Cause Failure Analysis: Fact or...

Documents

Transcript of Root Cause Failure Analysis: Fact or Fiction? Part 2 · Root Cause Failure Analysis: Fact or...