Auditing Risk Management

8/11/2019 Auditing Risk Management

1/41

Company Confidential

Registration Management Committee (RMC)

1

How to Audit Risk Management

Atlanta, GAJuly 22 & 23, 2010

Kimberly MaggieRon Tarach

QUAL-TECH, INC.

Auditor Workshop

Atlanta, GA

July 22-23, 2010


2/41


Atlanta, GA

July 22-23, 2010 2

Agenda

What is Risk?

Risk Management Process

Examples Risk Management Criteria

Auditor perceptions of Risk Management

Risk Management Tools

Auditor knowledge of tools and actions


3/41


Atlanta, GA

July 22-23, 2010 3

Agenda (continued)

Audit Planning

Audit Planning Tools

Activity 1 - Brainstorming session using

Audit Planning Tool

Conducting the Audit of Risk ManagementProcess

Examples of areas to evaluate

Activity 2 - Brainstorming session using CaseStudy and Failure Modes and Effects Analysis(FMEA)


4/41


Atlanta, GA

July 22-23, 2010 4

Ice Breaker!


5/41


Atlanta, GA

July 22-23, 2010 5

What is Risk?

An undesirable situation or circumstance thathas both a likelihood of occurring and a

potentially negative consequence.

AS9100:2009, clause 3.1


6/41


Atlanta, GA

July 22-23, 2010 6

Risk is inherent in all processes. Unfortunately, we dontsee the results of ineffective risk management methods

until later.


7/41


Atlanta, GA

July 22-23, 2010 7

Risk Management Process

Most organizations spend a great deal of time andmanpower trying to document Risks but manytimes this data is decentralized and not easilyaccessible to the functions that need thisinformation.

Process manufacturing can be so complex thatRisks can be very subtle and if there is not astructured Risk Management Process that takesadvantage of corporate knowledge, lessons learnedan organizations exposure to Risk can remain

high.


8/41


Atlanta, GA

July 22-23, 2010 8


9/41


Atlanta, GA

July 22-23, 2010 9

Examples of Risk Management Criteria

Understanding the types of risk that could comeinto a company. They could be related to

Employees

Process

Design Manufacturing

Equipment

Environment

Project Security


10/41


Atlanta, GA

July 22-23, 2010 10


Understanding the types of risk that could comeinto a company cont.

External

Contractor


11/41


Atlanta, GA

July 22-23, 2010 11


(continued)

Employees the organizations need toensure the safety, training, andqualifications of employees.

Process managing process variation.

Design building quality into the productdesign from the start, including its affect

on planning. Manufacturing ensuring that

manufacturing is more efficient withstreamlined quality planning.


12/41


Atlanta, GA

July 22-23, 2010 12

Criteria for Risk Management Process

(continued) Equipment ensuring that equipment can

meet capabilities, current and future.

Environment ensuring that theoperations are not compromising theenvironment (adequate lighting,temperature control, noise, cleanliness,etc).

Security managing the security neededby the facility.

Project ensuring project risks areevaluated before beginning.


13/41


Atlanta, GA

July 22-23, 2010 13

Criteria for Risk Management Process

(continued) External developing plans to address the

potential impact of weather, issues withtransportation companies, city

infrastructure (relating to construction,road closures).

Contractor ensuring impact is consideredfor contractors working on the building,

equipment, or with employees.


14/41


Atlanta, GA

July 22-23, 2010 14

Auditor Perceptions of Risk Management Thats the way we identified and handled risk

when I worked at Aviation Anywhere, Inc.

When I audited a Original EquipmentManufacturer (OEM) last month they wereusing FMEAs.

This little company only uses tool XYZ they

cant be managing risk properly.


15/41


Atlanta, GA

July 22-23, 2010 15

Auditor Perceptions of Risk Management(continued)

Remember, the design and implementationof an organizations aerospace quality

management system is influenced by varyingneeds, particular objectives, the productsprovided, the processes employed and thesize and structure of the organization.

AS9100:2009 General


16/41


Atlanta, GA

July 22-23, 2010 16

Auditor Perceptions of Risk Management(continued)

Organizational application of Risk can varybased on situation, customer, product line.

Audit approach & interviewing will need to beappropriate to the organization.

Remember, what is Appropriate to the

organization.


17/41


Atlanta, GA

July 22-23, 2010 17


18/41


Atlanta, GA

July 22-23, 2010 18


FMEAs e.g. dFMEA, pFMEA, etc.

Fault Tree Analysis (FTA)

Probabilistic Risk Assessment (PRA)

Event Tree Analysis (ETA)

Event Sequence Diagram (ESD)

Master Logic Diagrams (MLD)

Reliability Block Diagram (RBD)


19/41


Atlanta, GA

July 22-23, 2010 19

Risk Management Tools (continued)

Risk Assessment Matrix

Likeliness/Consequence Table

SWOT (Strength Weakness Opportunity

Threat) Business Continuity/Current Capability

Matrix

Risk Map and Control Scale


20/41


21/41


Atlanta, GA

July 22-23, 2010 21

Risk controlled or Oh No?


22/41


Atlanta, GA

July 22-23, 2010 22

Risk Management Tools (FMEA)


23/41


Atlanta, GA

July 22-23, 2010 23

Risk Management Tools (Influencer Analysis)


24/41


Atlanta, GA

July 22-23, 2010 24

Risk Management Tools (Risk Consequence)


25/41


Atlanta, GA

July 22-23, 2010 25



26/41


Atlanta, GA

July 22-23, 2010 26

Audit Planning

Selecting the right audit tool.

Identifying your audit criteria and anyreference documents.

Identifying your audit scope, includingidentification of the organizational andfunctional units and processes to beaudited.

Identifying an appropriate audit scope.


27/41


Atlanta, GA

July 22-23, 2010 27

Audit Planning Tools

Process (Turtle) Tool

Process Map Tool

Supplier Input Process Output Customer(SIPOC) Form

Process Based Management (PBM) ProcessFlow


28/41


Atlanta, GA

July 22-23, 2010 28

Process (Turtle) Tool

With What

(Materials, Equipment, Facilities)

Inputs (information and

material from other

processes)

How?

(Methods/Procedures/Techniques

With Who?

(Comp./Skills/Training)

Outputs (information

and Material to other

processes

How Effective/Efficient?

(Measurable Objective)

Process


29/41


Atlanta, GA

July 22-23, 2010 29

Process Map


30/41


Atlanta, GA

July 22-23, 2010 30

Supplier Input Process Output Customer(SIPOC) Form


31/41


Atlanta, GA

July 22-23, 2010 31

Process Based Management (PBM) ProcessFlow


32/41


Atlanta, GA

July 22-23, 2010 32


Audit Planning Tool


33/41


Atlanta, GA

July 22-23, 2010 33

Process (Turtle) Tool (Design)With What

Risk Management Software

Forms

Documents

Inputs

Customer, Internal Organization,Regulatory, Statutory

Special Requirements (e.g. product or

process complexity)

Critical Items (functions, parts, software,

characteristics, processes)

How?

AS9100, AS9110 and AS9120 Standards

Quality Manual

Standard Operating Procedure for Contracts

FMEA


With Who?

Sales

Engineering

Production

Quality

Outputs

Design

Planning

Production

Purchasing

Suppliers

Shipping


Customer complaints

In process/final rejection

Design verification/validation

Process

Contract Review

- Risk Management

Outputs

Drawing/Spec

Travelers

Routers

Work Orders

Inspection Reports


34/41


Atlanta, GA

July 22-23, 2010 34

Process (Turtle) Tool (Design Excluded)With What

Risk Management Software

Forms

Documents

Inputs

Customer, Internal Organization,Regulatory, Statutory

Special Requirements (e.g. product or

process complexity)

Critical Items (functions, parts, software,

characteristics, processes)

How?

AS9100, AS9110 and AS9120 Standards

Quality Manual

Standard Operating Procedure for Contracts

FMEA


With Who?

Sales

Engineering

Production

Quality

Outputs

Planning

Production

Purchasing

Suppliers

Shipping


Customer complaints

In process rejection

Final rejection

Process

Contract Review

- Risk Management

Outputs

Travelers

Routers

Work Orders

Inspection Reports


35/41


Atlanta, GA

July 22-23, 2010 35


Examples of areas to evaluate

Are all Risk identified during the RFQ and ContractReview Process e.g. special requirements, criticalrequirements.

Ensure Top management clearly understands whatRisks they have and what they are doing to ensurethey are mitigating those Risk.

Evaluate the selected Risk Management Tool foreffectiveness.

How are Risks communicated and managedthroughout the organization e.g. Design, Planning,Purchasing, Suppliers, Manufacturing, Inspection,Delivery and Post Delivery.

Design inputs, Design FMEAs, Design Verification andValidation.


36/41


Atlanta, GA

July 22-23, 2010 36


Examples of areas to evaluate continued

Critical characteristics across the quality lifecycle,ensuring the Process FMEAs and Control Plans arelinked.

Processes in place for capturing leading and laggingindicators related to Design Quality Performance.

Evaluate whether the organization has closed loopContinual Improvement Processes that captures andsustains Product and Process Quality.

Organization is using Lessons Learned and BestPractices.


37/41


Atlanta, GA

July 22-23, 2010 37


Examples of areas to evaluate continued

Ensure organizations Change Management Processinvolves the right people at the right time with theright process.

Ensure integration of Change Management withassessments to ensure correct consideration of Risk.

Ensure Risk Assessment tracked, recommendedcontrols to completion and ensured that Risk weremitigated as prescribed.

Ensure controls are in place for Risk that still remainafter mitigation actions.


38/41


Atlanta, GA

July 22-23, 2010 38


Case Study and FMEA


39/41


Atlanta, GA

July 22-23, 2010 39

Closing!


40/41


Atlanta, GA

July 22-23, 2010 40

Questions!


41/41


G

References

1.AS9100:2009

2.ISO 19011

3.FAA Risk Management Handbook 2009

4.NASA

Auditing Risk Management

Documents

Transcript of Auditing Risk Management