Risk-Based Thinking and

the Supplier Audit

Lance Coleman, Sr., ASQ CQE/SSGB/CQA/CBA, Exemplar Global QMS-PR

Kristen Wagner, B.S. Materials Science and Engineering

ASQ Section 0606 Meeting - September 13, 2018

❑ ASQ Senior Member, CQE, CSSGB, CQA, CBA

❑ 2018 ASQ Lean Enterprise Division Chair

❑ ASQ Instructor for Certified Quality Auditor Exam Preparatory Course

❑ Exemplar Global Principle QMS Auditor

❑ 2016-2017 Chair, US TAG 302 –Auditing Management Systems

❑ Voting Member US TAG 176 – Quality Assurance/Quality Management

❑ AAS EET, Southern Polytechnical University

❑ Author, Managing Organizational Risk Using the Supplier Audit Program

(Quality Press 2018)

❑ Author, Advanced Quality Auditing: An Auditor’s Review of Risk

Management, Lean Auditing and Data Analysis (Quality Press 2015)

Lance B. Coleman

(C) Lance Coleman2

❑ Supplier Quality Engineer II at Boston Scientific

❑ B.S. in Materials Science and Engineering from the

University of Minnesota – Twin Cities

❑ Social Media Chair for the Society of Women Engineers –

Minnesota Section

Kristen Wagner

3

Learning Objectives

1. How Risk Based Thinking Leads to Risk

Management

2. Supply Chain Management Overview

3. Supplier Auditing Fundamentals

4. Risk Based Supplier Auditing

5. Useful Tools

• 3-Keys to Asking Good Questions

• External Provider Audit Decision Trees

• Supplier Audit Preparation Checklist

Defining Risk

❑ISO 14971:2007– combination of the

probability of occurrence of harm and the

severity of harm

❑ISO 31000:2009 – the effect of uncertainty

on objectives

❑ISO 9001:2015 – the effect of uncertainty

Note: Hazards are things in an environment

(nouns) that have some risk attached to them

Classifying Risk

❑Consumer Risk – the risk of accepting a

bad part as good

oHarm or injury

oMalfunction

oNot meeting requirements

❑ Producer Risk – the risk of rejecting a good

part as bad

o Part costs

o Other failure costs

For want of a nail

ISO 9001:2015:6.1.1

When planning for the quality management

system, the organization shall consider the

issues...and determine the risks and

opportunities...

“Not all change is improvement but all

improvement is change”

Chuck Anger

V.P. Operations

Ultradent Products Inc.

ISO 9001:2015 Increased

Emphasis on Risk-Based Thinking

❑There is no requirement for formal methods

for risk management or a documented risk

management process in the standard

❑ Organizations can determine how to best

address and manage risk for their needs

❑ An Organizations’ risk management program

should be constantly improving

Risk Based Thinking

Risk-based thinking enables an organization to

determine the factors that could cause its

processes and its quality management system

to deviate from the planned results, to put in

place preventative controls to minimized

negative effects and to make maximum use of

opportunities as they arise

ISO 9001:2015 - 0.1 General

Risk Based Thinking

PROCESS INPUTS RISK MANAGEMENTRisk BasedThinking

- Inspection data

- Audit findings

- Management Review

- Test data

- Continuous Improvement

- Operator feedback

- Risk Model

- Risk Management Plan

- Reporting structure

- Feedback loops

What is an audit?

Results of the evaluation of the collected audit

evidence against audit criteria

ISO 19011:2018

Systematic, independent, and documented

process for obtaining audit evidence and

evaluating it objectively to determine the extent to

which audit criteria are fulfilled

ASQ CQA Handbook, Fourth Edition

And a supplier audit...

Onsite [supplier] verification activity, such as

inspection or examination, of a process or quality

system to ensure compliance to requirements…

The ASQ CSQP Handbook, First Edition

Key purpose is to confirm…

❑ Continued ability to meet product requirements

❑ Continued ability to meet production demands

❑ Maintenance of an effective quality

management system

❑ Maintenance of positive supplier relationship

Supplier vs. Internal Audit

❑Less process knowledge

❑Less visibility

❑Less control

❑Planning lead time

Supplier Audit Process

❑ Selection of Suppliers to Audit

o Supplier Assessment

o Supply Chain Management Feedback Loop

❑ Planning Audit

o Purpose, Scope, Resources

❑ Conducting Audit

❑ Report Write-Up

❑ Audit Follow-Up and Closure

Selection of Suppliers to Audit

❑ New or current supplier?

❑ Supplier assessments including surveys

❑ Supplier performance management

including scorecards

❑ Supply Chain Management feedback loop

Supplier Survey

❑ Should cover cross-functional areas

oQMS, Facilities, Engineering, R&D, Etc.

❑ Should include a section for key

personnel contact information

❑ Should be used to select areas of

focus for the on-site audit

Supply Chain Management

Supplier Performance Risk Based Decision

Feedback

Supplier

Management- Audits

- Inspection/testing

- Document Review

- Data Collection

- Reporting

(c) 2012 Lance Coleman

- Supplier risk classification

- Supplier Quality

- On-Time-Delivery

- Continue supplier

- Downgrade supplier

- Upgrade supplier

- Adjust rating

- Discontinue supplier

Supplier Performance Monitoring

Tangibles

❑ Quality(e.g. - Nonconformances and Supplier Corrective Action Requests issued)

❑ On-Time-Delivery

❑ In-Full-Delivery

❑ Audit results

Supplier Performance Monitoring

Intangibles

❑ Supplier partnership

o Participation in Continuous Improvement,

Six Sigma, and/or Lean Projects

o Excellent service

o Easy to work with

❑ Responsive to questions or concerns

❑ Open and forthcoming with information

Applying Risk Based Thinking to

Audit Planning

1) Understand financial, time, and personnel

resource allocation

2) Perform risk assessment of supplier pool

o Creation of justification during exercise

3) Utilize risk assessment learnings to support

supplier selection and justification for more

resources as needed

Risk Assessment Inputs

❑ New or current supplier

❑ Single or multi-sourced product

❑ Supplier audit schedule

❑ Major observations or findings from previous audit

❑ Results of change impact assessment

o Significant change of business focus for supplier

o Acquisition

o New management

o High employee turnover

❑ Supplier Performance Monitoring results

o SCAR and NC frequency

o Deviations from procedures and/or specifications

o Poor communication

❑ Internal, external, or third-party rejections

Audit Schedule Decision Tree

Possibiliy of serious injury

Significant Performance Degradation

Possibility of Injury

Significant Business Risk

Performance Degradation

ISO Certified

ISO Certified

Somewhat to very likely

Onsite Audit Somewhat to very likely

Somewhat to very likely

Onsite Audit Onsite Audit

Cosmetic Issues

Somewhat to very likely

ISO Certified

Audit

Audit

No AuditNew Supplier/

Product

Y Y

N

Y Y

YY

N

N N

N N NN

N

N

N

N

Y

Y

Y Y

Y

Y

Y

If Everything Is Important, Then Nothing Is

Apply impact values to risk assessment inputs

based on…

❑ Company strategy

❑ Industry requirements

❑ Business needs

…to acquire final risk score per supplier

Backstory

❑ There are 3 suppliers that are being

considered for audits this year

o Supplier A, B, and C

❑ Resource restrictions dictate that you can

only select 2 suppliers to audit

Who do you choose?

Supplier Risk Matrix

Supplier A

❑ Current Classification A supplier that is due for an

audit this year

❑ There were 2 marginal findings during the last audit;

however they are easily corrected given supplier

dedication

❑ A new inspector was hired; however, the supplier

provided training records

❑The supplier has had numerous issues with on-time-

delivery in the past year

❑ With the implementation of a safety stock, the Supply

Chain team has been able to handle the on-time

delivery issues with minimal customer upsets;

however, upper management set a strategic company

goal to reduce inventory cost which requires a

reduction in safety stock

Supplier B

❑ Current Classification B supplier for a raw material with

several non-active alternative qualified suppliers

❑ Supplier not due for audit this year

❑ There were minor nonconformances during the last audit;

however, they were corrected the day of the audit

❑ A new inspector was hired; however, the supplier provided

training records

❑ There have been several instances of nonconforming

product that Incoming has captured. There have been

sporadic issues on the production floor connected to this

material that is causing product failures.

❑ Some product failures were not captured under final

inspection and made it’s way to a customer. The customer

sent the product to a third party organization to confirm the

failure.

Supplier C

❑ Supplier C is a new Classification A supplier

that is connected to a company top priority

project.

o New suppliers’ risk assessment should have their

own unique risk factor inputs based on intended

use and should involve project team collaboration

❑ No previous performance information is

known about supplier

❑ Material provided by supplier is a new

material to your company

Supplier AImpact Value Mitigation Value Risk Ranking

Supplier Classification 4 3 12

Due for Scheduled Audit 4 3 12

Major Findings from Previous Audit 3 2 6

Change Impact Assessment Results 1 1 1

Supplier Performance Monitoring Results 4 3 12

Internal Rejections 3 3 9

External Rejections 2 2 4

Third-Party Rejections 0 0 0

Final Risk Score:56

Supplier BImpact Value Mitigation Value Risk Ranking

Supplier Classification 2 2 4

Due for Scheduled Audit 1 1 1

Major Findings from Previous Audit 1 1 1

Change Impact Assessment Results 1 1 1

Supplier Performance Monitoring Results 4 3 12

Internal Rejections 4 4 16

External Rejections 4 3 12

Third-Party Rejections 4 3 12

Final Risk Score:59

Which suppliers should be audited?

❑ Supplier C

o Most Risk

o Unknown performance

o New material to company

o Connection to a company strategic project

o BUDGET SAVINGS! Apply audit expense under

budget for project

❑ Supplier B

o Even though Supplier A is due for an audit and

there is potential risk to meeting customer orders,

Supplier B has more realized risk per the risk

assessment

What about Supplier A?

JUSTIFICATION

Use learnings from risk assessment and other

supplier management tools, such as

performance monitoring, to build a case for

additional resources

Important Do’s and Don’ts

Do : demonstrate risk based thinking throughout

your QMS

Don’t: just sprinkle the word risk throughout your

documents with nothing to support the concept

Do : ask and answer the hard questions

Don’t: state that you will do something in order to

meet the requirement and then don’t do it

Do : include an appropriate amount of specificity

in your procedures

Don’t: write yourself into a corner or overuse or

imply phrases such as whenever, always, every

time

Supplier Audit Needs Decision Tree

Conducting Risk Based Audits

❑ Directly audit the risk management (RM)

program itself

❑ Conduct RBQA of aspects of the QMS or

of the QMS as a whole

o Standalone risk management audit of

QMS elements

o Incorporate risk management into

existing audits

Robust Risk Management

For a truly robust risk management program,

the following as a minimum should occur:

1. The program should encompass all aspects

of a product life cycle from design to end-of-

life disposal.

2. Data from external as well as internal

sources should be captured and analyzed

and the risk model updated as necessary.

3.Teams, when formed, should be cross-

functional in nature in order to model the

broadest range of risks

Auditing the RM Program

1. First, confirm all three of the items from the

previous slide are occurring

2. Confirm that results from the risk management

program are reported as necessary to

appropriate levels of management

3. Confirm that existing risk management

procedures and work instructions are followed

4. Ensure that organizational training supports the

risk management program

5. Confirm that adequate resources are supplied

to meet the goals of the risk management

program

Where does risk lie in our process?

1. Complexity of the process

2. Complexity of the product

3. Criticality of the product

4. Location where most processing has occurred

5. Newness of the product

6. Newness of employees

7. History of the process

Risky Behavior

How do we know risky behavior or situations

when we see them?

• Variance from industry norms

• Employee concerns

• Established feedback channels

• Identified in risk management plan

Risky Behavior

You will often know it when you see it...

Scenario: An auditor visits a machine shop

and witnesses a welding operation where

sparks are flying in the immediate vicinity. The

operator is following their instructions and

wearing the appropriate PPE, but the auditor

notices a small puddle of oil on the floor

nearby.

Identify the hazards, risks, and mitigations

(controls) in this scenario.

Identifying Hazards & Risks

3-Keys to Good Questioning

1. Ask the right question

2. Ask the obvious question

3. Let one question lead you to the next

Risk Based Thinking & Audit Findings

Key Thought:

In thinking about how risk affects the

classification of audit findings, we can look at

the risk of individual nonconformities, the risk

found within aspects of the quality management

system, or the risk found within the overall

quality management system

Risk Level Definitions - 1

RA-Significant Risk:

• Potential for product contamination, complete

product failure, or serious supply chain

disruption

• Potential violation of customs/regulatory

requirements or blatant disregard of

Technical/Quality Agreement

• Multiple systemic or chronic deviations from the

requirements

• Conformity required within three months or

according to agreement

Risk Level Definitions - 2

B-Moderate Risk:

• Required procedures do not exist or exist, but not

implemented or followed

• Lack of awareness or attention to cGMP requirements

• Lack of IT and other technical resources available and

appropriate to the size of the business

• Deficient management systems to handle Customer

Service, Production Planning, and Inbound/ Outbound

Logistics

• Significant number of instances of partial fulfillment of

requirements

• System is not achieving defined objectives

• Compliance required within six months or according to

agreement

Risk Level Definitions - 3

C-Minor Risk:

• Regulatory, ISO, contractual, and internal requirements

met in principle but not in full

• Current system needs additional focus and other

improvements

• Compliance required within six months or according to

agreement

Note: In this case, it is not each individual audit finding

that is assessed for risk, but rather the aggregate effect

of all audit nonconformities related to a particular

aspect of the quality management system and the risk

of those nonconformities causing a failure in the

system

QMS

Improperly

completed forms

and records

(Information still

retrievable)

Violation of

internal procedure

or work

instruction;

Current practice

that meets

requirement is not

accurately

documented

Violation of customer

requirement or

internal requirement.

Systemic or chronic

failure of QMS

requirement. Multiple

related minor

violations. Cause great

harm to other

operations in the

company.

Noncompliance

that is itself a

hazard or may

lead to hazardous

condition. Direct

violation of ISO

standards or

cGMP. Absense of

required procedure

or record

Impact Negligible

(1)

Minor

(2)

Major

(3)

Critical

(4)

Applying Risk Based Thinking to Audit Findings

Failure Likelihood Estimation Chart

LikelihoodProbability

RankDefinition

Very Low1

Unlikely to happen, rare, remote

Low2

Can happen, but not frequently

High3

Likely to happen, often, frequent

Very High4

Very likely to happen, more often than not

Applying Risk Based Thinking to Audit Findings

A

RISK

MATRIX

IMPACT

L

i

k

e

l

i

h

o

o

d

Negligible

(1)

Moderate

(2)

Marginal

(3)

Critical

(4)

Very Unlikely

(1)

Unlikely

(2)

Likely

(3)

Very Likely

(4)

Low Risk Medium Risk High Risk

Applying Risk Based Thinking to Audit Findings

Risk Based Audit Findings

Risk Level Low Medium High

Risk Level Description

Nonconformities that do not

affect form, fit or function.

Documentation errors that can

be fixed.

Regulatory noncompliances

other than those that could

cause injury, harm or

malfunction.

Product nonconformities

that may partially inhibit

function.

Noncompliances which could

cause harm or injury to end

users, distributors, company

employees or the public at

large.

Noncompliances that could

cause significant or total

functional failures.

Finding Classification Minor Minor/Major Major/Critical

“the single biggest problem in communication

is the illusion that it has taken place”

George Bernard Shaw

Audit Report Content - 1

A typical individual supplier audit report will include:

❑ Audit purpose and scope

❑ Audit criteria

❑ Lead Auditor and audit team members

❑ Summary of results

❑ Result details including identified risks

❑ Audit finding definitions/explanations

❑ Review of findings (nonconformities, opportunities for

improvement, and positive practices)

❑ Review of corrective actions

❑ Identification of the need (or not) for a follow-up audit

❑ Opportunities to improve for your organization

Audit Report Content - 2

The risk-based individual supplier audit report

will also address:

❑ Identified supplier risks and their

assessments

❑ Risk mitigations

❑ Residual risks

❑ Any risks attached to how your organization

conducts business with the supplier

Report Distribution

❑ Send report to supplier and open the floor

for discussion

❑Establish dates for any action items and

follow-up as necessary

❑ Discuss report with supplier stakeholders

❑Summarize risks and report to upper

management as needed

Key Takeaways

1. Applying risk based thinking to audits

allows us to ask better questions and

provide more critical review of data

2. When interviewing, it is important to ask

the correct and most obvious questions

3. Risk should be defined by each

organization according to their business

model

4. Following a structured approach for

supplier auditing will provide better results

ReferencesManaging Organizational Risk Using the

Supplier Audit Program

ISBN: 978-0-87389-968-0

Quality Press 2018

The Certified Supplier Quality Professional

Handbook

ISBN: 978-0-87389-943-7

Quality Press 2017

Advanced Quality Auditing

ISBN: 978-0-87389-913-0

Quality Press 2015

Performance Metrics:

The Levers for Process Management

Quality Press 2013

ISBN-13: 978-0873898508

Next Steps

❑ Assess your risk management program to

ensure that all types of risk are addressed

❑ Review your procedures to ensure

appropriate level of specificity and no

overuse of whenever, always, every and so

forth.

❑ Take ownership of closing identified gaps

Questions ??? Lance B. Coleman ASQ CQE/CSSGB/CQA/CBAQA&R ManagerIDEX Health and Science LLCSeattle, WA 98108360-682-4242Lcoleman@idexcorp.comhttps://www.linkedin.com/in/lance-b-coleman-asq-cqe-cssgb-cqa-cba-rabqsa-cqms-pr-7418131b/

Kristen WagnerSupplier Quality Engineer IIBoston ScientificMaple Grove, MN 55311KMWagner.MSE@gmail.comhttps://www.linkedin.com/in/kristen-wagner