How to Audit Risk Management
description
Transcript of How to Audit Risk Management
Company Confidential
Registration Management Committee (RMC)
1
How to Audit Risk Management
Atlanta, GAJuly 22 & 23, 2010
Kimberly MaggieRon Tarach
QUAL-TECH, INC.
Auditor WorkshopAtlanta, GA
July 22-23, 2010
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 2
Agenda• What is Risk?• Risk Management Process • Examples Risk Management Criteria• Auditor perceptions of Risk Management• Risk Management Tools
– Auditor knowledge of tools and actions
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 3
Agenda (continued)• Audit Planning
– Audit Planning Tools• Activity 1 - Brainstorming session using
Audit Planning Tool• Conducting the Audit of Risk Management
Process– Examples of areas to evaluate
• Activity 2 - Brainstorming session using Case Study and Failure Modes and Effects Analysis (FMEA)
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 4
Ice Breaker!
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 5
What is Risk?An undesirable situation or circumstance that
has both a likelihood of occurring and a potentially negative consequence.
AS9100:2009, clause 3.1
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 6
“Risk is inherent in all processes. Unfortunately, we don’t see the results of ineffective risk management methods
until later”.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 7
Risk Management Process – Most organizations spend a great deal of time and
manpower trying to document “Risks” but many times this data is decentralized and not easily accessible to the functions that need this information.
– Process manufacturing can be so complex that “Risks” can be very subtle and if there is not a structured “Risk Management Process” that takes advantage of corporate knowledge, lessons learned an organization’s exposure to “Risk” can remain high.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 8
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 9
Examples of Risk Management Criteria» Understanding the types of risk that could come
into a company. They could be related to• Employees• Process• Design• Manufacturing• Equipment• Environment• Project• Security
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 10
Examples of Risk Management Criteria» Understanding the types of risk that could come
into a company cont. • External• Contractor
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 11
Examples of Risk Management Criteria (continued)
– Employees – the organizations need to ensure the safety, training, and qualifications of employees.
– Process – managing process variation.– Design – building quality into the product
design from the start, including it’s affect on planning.
– Manufacturing – ensuring that manufacturing is more efficient with streamlined quality planning.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 12
Criteria for Risk Management Process (continued)
– Equipment – ensuring that equipment can meet capabilities, current and future.
– Environment – ensuring that the operations are not compromising the environment (adequate lighting, temperature control, noise, cleanliness, etc).
– Security – managing the security needed by the facility.
– Project – ensuring project risks are evaluated before beginning.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 13
Criteria for Risk Management Process (continued)
– External – developing plans to address the potential impact of weather, issues with transportation companies, city infrastructure (relating to construction, road closures).
– Contractor – ensuring impact is considered for contractors working on the building, equipment, or with employees.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 14
Auditor Perceptions of Risk Management• That’s the way we identified and handled risk
when I worked at Aviation Anywhere, Inc.• When I audited a Original Equipment
Manufacturer (OEM) last month they were using FMEAs.
• This little company only uses tool XYZ – they can’t be managing risk properly.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 15
Auditor Perceptions of Risk Management (continued)
“Remember, the design and implementation of an organization’s aerospace quality management system is influenced by varying needs, particular objectives, the products provided, the processes employed and the size and structure of the organization.”
AS9100:2009 General
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 16
Auditor Perceptions of Risk Management (continued)
• Organizational application of Risk can vary based on situation, customer, product line.
• Audit approach & interviewing will need to be appropriate to the organization.
• Remember, what is “Appropriate” to the organization.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 17
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 18
Risk Management Tools– FMEAs e.g. dFMEA, pFMEA, etc.– Fault Tree Analysis (FTA)– Probabilistic Risk Assessment (PRA)– Event Tree Analysis (ETA)– Event Sequence Diagram (ESD)– Master Logic Diagrams (MLD)– Reliability Block Diagram (RBD)
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 19
Risk Management Tools (continued)– Risk Assessment Matrix– Likeliness/Consequence Table– SWOT (Strength Weakness Opportunity
Threat)– Business Continuity/Current Capability
Matrix– Risk Map and Control Scale
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 20
Risk Management Tools (continued)– Auditor knowledge of tools and actions
» No one auditor has experience with all the tools available in the industry and how they are used.
» Familiarize your self with the various Risk Management Tools (self study).
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 21
Risk controlled – or “Oh No”?
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 22
Risk Management Tools (FMEA)
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 23
Risk Management Tools (Influencer Analysis)
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 24
Risk Management Tools (Risk Consequence)
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 25
Risk Management Tools
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 26
Audit Planning– Selecting the right audit tool.– Identifying your audit criteria and any
reference documents.– Identifying your audit scope, including
identification of the organizational and functional units and processes to be audited.
– Identifying an appropriate audit scope.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 27
Audit Planning Tools– Process (Turtle) Tool– Process Map Tool– Supplier Input Process Output Customer
(SIPOC) Form– Process Based Management (PBM) Process
Flow
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 28
Process (Turtle) Tool With What
(Materials, Equipment, Facilities)
Inputs (information and
material from other
processes)
How?
(Methods/Procedures/Techniques
With Who?
(Comp./Skills/Training)
Outputs (information
and Material to other
processes
How Effective/Efficient?
(Measurable Objective)
Process
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 29
Process Map
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 30
Supplier Input Process Output Customer (SIPOC) Form
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 31
Process Based Management (PBM) Process Flow
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 32
Activity 1 - Brainstorming session using Audit Planning Tool
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 33
Process (Turtle) Tool (Design) With What
Risk Management Software
Forms
Documents
Inputs
Customer, Internal Organization, Regulatory, Statutory
Special Requirements (e.g. product or process complexity)
Critical Items (functions, parts, software, characteristics, processes)
How?
AS9100, AS9110 and AS9120 Standards
Quality Manual
Standard Operating Procedure for Contracts
FMEA
Risk Assessment Matrix
With Who?
Sales
Engineering
Production
Quality
Outputs
Design
Planning
Production
Purchasing
Suppliers
Shipping
How Effective/Efficient?
Customer complaints
In process/final rejection
Design verification/validation
Process
Contract Review
- Risk Management
Outputs
Drawing/Spec
Travelers
Routers
Work Orders
Inspection Reports
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 34
Process (Turtle) Tool (Design Excluded) With What
Risk Management Software
Forms
Documents
Inputs
Customer, Internal Organization, Regulatory, Statutory
Special Requirements (e.g. product or process complexity)
Critical Items (functions, parts, software, characteristics, processes)
How?
AS9100, AS9110 and AS9120 Standards
Quality Manual
Standard Operating Procedure for Contracts
FMEA
Risk Assessment Matrix
With Who?
Sales
Engineering
Production
Quality
Outputs
Planning
Production
Purchasing
Suppliers
Shipping
How Effective/Efficient?
Customer complaints
In process rejection
Final rejection
Process
Contract Review
- Risk Management
Outputs
Travelers
Routers
Work Orders
Inspection Reports
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 35
Conducting the Audit of Risk Management Process
– Examples of areas to evaluate» Are all “Risk” identified during the RFQ and Contract
Review Process e.g. special requirements, critical requirements.
» Ensure Top management clearly understands what “Risks” they have and what they are doing to ensure they are mitigating those “Risk”.
» Evaluate the selected Risk Management Tool for effectiveness.
» How are “Risks” communicated and managed throughout the organization e.g. Design, Planning, Purchasing, Suppliers, Manufacturing, Inspection, Delivery and Post Delivery.
» Design inputs, Design FMEAs, Design Verification and Validation.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 36
Conducting the Audit of Risk Management Process
– Examples of areas to evaluate continued» Critical characteristics across the quality lifecycle,
ensuring the Process FMEAs and Control Plans are linked.
» Processes in place for capturing leading and lagging indicators related to Design Quality Performance.
» Evaluate whether the organization has closed loop Continual Improvement Processes that captures and sustains Product and Process Quality.
» Organization is using Lessons Learned and Best Practices.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 37
Conducting the Audit of Risk Management Process
– Examples of areas to evaluate continued» Ensure organization’s Change Management Process
involves the right people at the right time with the right process.
» Ensure integration of Change Management with assessments to ensure correct consideration of “Risk”.
» Ensure “Risk Assessment” tracked, recommended controls to completion and ensured that “Risk” were mitigated as prescribed.
» Ensure controls are in place for “Risk” that still remain after mitigation actions.
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 38
Activity 2 - Brainstorming session using Case Study and FMEA
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 39
Closing!
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 40
Questions!
Registration Management Committee (RMC)
Atlanta, GAJuly 22-23, 2010 41
References1.AS9100:20092.ISO 190113.FAA Risk Management Handbook 20094.NASA