Post on 06-May-2018
Capability Assessments for Continued
Process Verification: Overcoming
potential obstacles and common
pitfalls
SIV 2015
Presenter: Peter M. Saama, Ph.D.
Bayer HealthCare LLC
Agenda/
Content Page 1
Continued Process Verification (Phase III)
Overview: FDA Guidance, Process Capability
Small Sample Size
Non-Normality
Auto-correlation
Mean Shifts
Chronological Order
Round-off Errors
Random Fluctuations
Requirements for FDA Validation Guidance
• FDA Guidance for Industry: Process Validation: General Principles and Practices,
published January 2011 distinguishes three stages of validation
• Stage 1 – Process Design: The commercial manufacturing process is defined during
this stage based on knowledge gained through development and scale-up activities.
• Stage 2 – Process Qualification: During this stage, the process design is evaluated to
determine if the process is capable of reproducible commercial manufacturing.
• Stage 3 – Continued Process Verification: Ongoing assurance is gained during
routine production that the process remains in a state of control.
• Should understand sources of variation
• Detect the presence and degree of variation
• Understand the impact of variation on the process and, ultimately, product
attributes
• Control the variation in a manner commensurate with the risk it presents to the
process and product
• Control Chart depicts inherent variation in a process Stable (In-Control)
• Process Capability Index (PCI) assesses whether or not a process is capable of meeting
customer requirements (a.k.a. PCR)
• Cp, Cpk, Cpm Process is stable (In-Control)
Where with ; from Shewhart’s Tables; is the standard
deviation and T is the target value. Here Cpm is the so-called the Taguchi Index.
• Ppk Process with known and predictable special causes
• Where
Overview: Process Capability
Scope:
• Assume that:
• Critical Quality Attributes (CQA) have been identified for the product
• Parameter Criticality has been established for the process
• Critical Process Parameters (CPP) are linked to CQA
• Alert and action limits have been set for CPP
• Risk analysis was used to determine the monitoring and sampling requirements of
CPP’s and CQA’s
• CPV Strategy is fully implemented
• Examine Capability Assessments for commercialized products
• Small Sample Size (n < 30)
• Process Complexity
Product X Dataset: n = 50
Serial
No.
BULK
Batch
Batch
Disposition
Manufacturing
Date
% Label Claim
(90-110)%
1 PX9999R REL 8-Oct-2012 98.8
2 PX99020 REL 15-Oct-2012 98.4
3 PX990WR REL 24-Oct-2012 98.6
4 PX990WT REL 15-Nov-2012 98.4
5 PX990WV REL 16-Nov-2012 97.4
6 PX990WW REL 10-Dec-2012 99.2
7 PX990QW REL 12-Dec-2012 99.3
8 PX990NV REL 14-Dec-2012 99.9
9 PX990RW REL 17-Dec-2012 99.8
10 PX991WW REL 21-Dec-2012 100.4
11 PX991WX REL 10-Jan-2013 98.4
12 PX991V0 REL 11-Jan-2013 98.2
13 PX991V1 REL 25-Mar-2013 98.4
14 PX991V2 REL 1-Apr-2013 98.6
15 PX992WN REL 11-Apr-2013 96.8
16 PX993N1 REL 12-Apr-2013 98.2
17 PX994RW REL 18-Apr-2013 98.1
18 PX994RR REL 25-Apr-2013 98.4
19 PX9954M REL 12-Jun-2013 96.8
20 PX995WR REL 19-Jun-2013 96.6
21 PX99676 REL 20-Jun-2013 97.3
22 PX99677 REL 9-Aug-2013 97.8
23 PX997VM REL 9-Aug-2013 97.0
24 PX997VN REL 21-Aug-2013 98.4
25 PX997VP REL 30-Aug-2013 98.2
Serial
No.
BULK
Batch
Batch
Disposition
Manufacturing
Date
% Label Claim
(90-110)%
26 PX998T0 REL 16-Oct-2013 101.1
27 PX998MV REL 11-Nov-2013 100.7
28 PX998T1 REL 14-Nov-2013 100.3
29 PX998T2 REL 21-Nov-2013 100.6
30 PX999NR REL 27-Jan-2014 98.1
31 PX99WRV REL 27-Jan-2014 98.9
32 PX999NP REL 3-Feb-2014 98.5
33 PX99WR9 REL 10-Feb-2014 100.1
34 PX99WRW REL 10-Feb-2014 98.2
35 PX9996M REL 17-Feb-2014 97.2
36 PX99RM8 REL 19-Feb-2014 98.6
37 PX99Q35 REL 10-Apr-2014 96.9
38 PX99Q36 REL 11-Apr-2014 96.9
39 PX99O0W REL 17-Apr-2014 97.7
40 PX99O0X REL 21-Apr-2014 97.9
41 PX99O7R REL 28-Apr-2014 98.4
42 PX99OV9 REL 9-Jun-2014 97.2
43 PX99OV8 REL 16-Jun-2014 97.4
44 PX99OXM REL 19-Jun-2014 99.2
45 PX99NQ9 REL 7-Jul-2014 94.4
46 PX99PM0 REL 9-Jul-2014 97.8
47 PX99N76 REL 28-Jul-2014 98.7
48 PX998MT REL 27-Aug-2014 102.2
49 PX99R9X REL 15-Sep-2014 100.8
50 PX99PXR REL 13-Oct-2014 99.1
Small Sample Size: n < 30
• SPC Assumptions Normality, Independence/Exchangeability, Chronological order,
Specifications based on consumer’s risk, zero autocorrelation, n ≥ 30.
• Impact: Conventional SPC methods (Cp, Cpk) do not apply
• First 25 batches (Product X)
Serial
No.
BULK
Batch
Batch
Disposition
Manufacturing
Date
% Label Claim
(90-110)%
1 PX9999R REL 8-Oct-2012 98.8
2 PX99020 REL 15-Oct-2012 98.4
3 PX990WR REL 24-Oct-2012 98.6
4 PX990WT REL 15-Nov-2012 98.4
5 PX990WV REL 16-Nov-2012 97.4
6 PX990WW REL 10-Dec-2012 99.2
7 PX990QW REL 12-Dec-2012 99.3
8 PX990NV REL 14-Dec-2012 99.9
9 PX990RW REL 17-Dec-2012 99.8
10 PX991WW REL 21-Dec-2012 100.4
11 PX991WX REL 10-Jan-2013 98.4
12 PX991V0 REL 11-Jan-2013 98.2
13 PX991V1 REL 25-Mar-2013 98.4
14 PX991V2 REL 1-Apr-2013 98.6
15 PX992WN REL 11-Apr-2013 96.8
16 PX993N1 REL 12-Apr-2013 98.2
17 PX994RW REL 18-Apr-2013 98.1
18 PX994RR REL 25-Apr-2013 98.4
19 PX9954M REL 12-Jun-2013 96.8
20 PX995WR REL 19-Jun-2013 96.6
21 PX99676 REL 20-Jun-2013 97.3
22 PX99677 REL 9-Aug-2013 97.8
23 PX997VM REL 9-Aug-2013 97.0
24 PX997VN REL 21-Aug-2013 98.4
25 PX997VP REL 30-Aug-2013 98.2
10810510299969390
LSL USL
LSL 90
Target *
USL 110
Sample Mean 98.3024
Sample N 25
StDev (Within) 0.617612
StDev (O v erall) 0.974206
Process Data
C p 5.40
C PL 4.48
C PU 6.31
C pk 4.48
Pp 3.42
PPL 2.84
PPU 4.00
Ppk 2.84
C pm *
O v erall C apability
Potential (Within) C apability
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
O bserv ed Performance
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
Exp. Within Performance
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
Exp. O v erall Performance
Within
Overall
Process Capability of % Label Claim
Small Sample Size: n < 30
-Suggested Solutions-
• For CPP/CQA with Two-Sided limits, estimate Two-Sided 95% or 99% or 99.9% Tolerance
Intervals
• Note: Risk Assessment is used to define, apriori, the uncertainty (99.9% or 99% or
95%) as part of the PROCESS CONTROL STRATEGY.
• 99% (p=.99) of the population with confidence level of 95% (α=0.05; γ=0.95)
consistent with standard coverage limits for SPC
• Assume 99% uncertainty and 95% confidence
• Estimate Tolerance Factor, k2, according to Guenther (1977)
, i.e.,
• Tolerance Interval is:
• Use Lower Tolerance Bound (LTB) and Upper Tolerance Bound (UTB) estimates
from 99% Tolerance Interval.
• For a single sample, use the Cpk metric:
Small Sample Size: n < 30
-Suggested Solutions-
• For CPP/CQA with a one-sided limit, estimate a one-sided 95% or 99% or 99.9%
Tolerance Intervals
• Assume 99% uncertainty (p=.99) and 95% confidence (α=0.05; γ=0.95)
• Estimate Tolerance Factor, k1, according to Natrella (1963):
, i.e.,
• Tolerance Interval is:
• Use Lower Tolerance Bound (LTB), , and Upper Tolerance Bound (UTB)
, , estimates from 99% Tolerance Interval.
• For a single sample, use the CpL and CpU metrics:
Small Sample Size: n < 30
-Suggested Solutions-
• Generate 15 ‘batches’ from the Normal Distribution, N(1.004, 0.01332): Product U
• Generate 30 ‘batches’ from the Normal Distribution, N(1.004, 0.01332): Product W
Small Sample Size: n < 30
-Suggested Solutions-
• Capability assessment for Product U using small-sample approximation,
• Capability assessment for Product U using the standard analysis:
Small Sample Size: n < 30
-Suggested Solutions-
• Capability assessment for Product W using small-sample approximation,
• Capability assessment for Product W using the standard analysis:
Small Sample Size: n < 30
-Suggested Solutions-
• For multiple samples, use the Ppk metric
• As noted previously, use half-width of the Tolerance Interval in the divisor(s) for the
Process Capability Index (PCI)
• 99% Population Proportion
• 95% Confidence Level
• For Risk Assessment (FMEA): RPN=v, use α=c=.05, Tolerance Interval= [LTB, UTB]
Process Complexity (Issue 1): Non-Normality
• Evidence of Non-normality in Histogram plot
• Impact: Unreliable estimates of Process Capability Indices
Process Complexity (Issue 1): Non-Normality
-Suggested Solutions-
• Probability Plot:
where is the 99.865 percentile (see below)
• Johnson Transformation followed by ‘Normal’ Process Capability
where is a standard normal variate, are unknown, and
Lognormal parameter estimates:
where 100αth percentile is α(n+1)th-ranked value from n observations.
Process Complexity (Issue 1): Non-Normality
-Suggested Solutions-
• Box-Cox Transformation followed by Normal Process Capability
where is obtained by Maximum Likelihood Estimation
• ‘Non-Normal’ Process Capability
• Log-Normal
• Error does not depend on measurement
• Constant Standard Deviation
• Process is centered
• Log-Transformation followed by ‘Normal’ Process Capability
Process Complexity (Issue 1): Non-Normality
-Suggested Solutions-
• Performance under slight, moderate, and severe departures from Normality
• PCI values (Cpu) from lognormal distribution with µ=0.0 and σ2=0.5 (n=100).
• PCI values (Cpu) from lognormal distribution with µ=0.0 and σ2=0.3 (n=100).
• Box-Cox and Johnson transformations are superior (n ≥100)
• Not appropriate for small sample sizes; unintentional process shift is induced.
• Probability Plot is suitable when there is a mild departure from normality for the underlying
process.
Process Complexity (Issue 2): Auto-Correlation
• High association between adjacent data points. Caused by:
• Production Lot
• Production (Weeks)
• Raw Material Lot (early in production cycle)
• Impact: Profound effect on Control Chart and Process Capability.
• Cpk underestimate total process variation
• Positive autocorrelation more signals in Control Chart
• In-control ARL becomes shorter
Process Complexity (Issue 2): Auto-Correlation
-Suggested Solution(s)-
• Short-term limits = 6 * short-term Sigma
• Long-term limits= 6 * long-term Sigma
• Compare Cpk to Ppk
• Ppk (Long Term ) more representative of Process Capability
• Monitor autocorrelation structure using Sample autocorrelation function (ACF), e.g.
Product X
13121110987654321
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
Lag
Au
toco
rre
lati
on
Autocorrelation Function for % Label Claim(with 5% significance limits for the autocorrelations)
Process Complexity (Issue 2): Auto-Correlation
-Suggested Solution(s)-
• Less frequent sampling can remove autocorrelation
• Product X every 2nd observation (sub-sampling): CAF
654321
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
Lag
Au
toco
rre
lati
on
Autocorrelation Function for % Label Claim(with 5% significance limits for the autocorrelations)
Process Complexity (Issue 3): Mean-Shifts
• Inherent due to:
• Campaign effects
• Raw Material Changes
• Seasonal effects on the manufacturing process
• Scale-up (e.g. Larger Tank or Increase in Batch Size)
• Affects Critical Process Parameters (CPP): e.g. Mixing Speeds
• Affects Critical In-Process Controls: Net Yield, Yield %
• Increased Risk of Operator Error
Process Complexity (Issue 3): Mean-Shifts
-Suggested Solution(s)-
• Compare Cpk to Ppk
• Ppk (Long Term) more representative of Process Capability
• Scale-up/Scale-down
• Data Sub-groups Compounding Tank / Batch Size
• Capability Analysis w/n sub-group (n ≥ 30)
• If Cpk ≥ 1.33 in all sub-groups Process is Capable
• CUSUM / EWMA charts to detect mean shifts
PX99PXR
PX99R9X
PX998MT
PX99N76
PX99PM0
PX99NQ9
PX99OXM
PX99OV8
PX99OV9
PX99O7R
PX99O0X
PX99O0W
PX99Q36
PX99Q35
PX99RM8
PX9996M
PX99WRW
PX99WR9
PX999NP
PX99WRV
PX999NR
PX998T2
PX998T1
PX998MV
PX998T0
PX997VP
PX997VN
PX997VM
PX99677
PX99676
PX995WR
PX9954M
PX994RR
PX994RW
PX993N1
PX992WN
PX991V2
PX991V1
PX991V0
PX991WX
PX991WW
PX990RW
PX990NV
PX990QW
PX990WW
PX990WV
PX990WT
PX990WR
PX99020
PX9999R
99.5
99.0
98.5
98.0
97.5
BULK Batch
EW
MA
__X=98.486
UCL=99.433
LCL=97.538
EWMA Chart of % Label Claim
PX99PXR
PX99R9X
PX998MT
PX99N76
PX99PM0
PX99NQ9
PX99OXM
PX99OV8
PX99OV9
PX99O7R
PX99O0X
PX99O0W
PX99Q36
PX99Q35
PX99RM8
PX9996M
PX99WRW
PX99WR9
PX999NP
PX99WRV
PX999NR
PX998T2
PX998T1
PX998MV
PX998T0
PX997VP
PX997VN
PX997VM
PX99677
PX99676
PX995WR
PX9954M
PX994RR
PX994RW
PX993N1
PX992WN
PX991V2
PX991V1
PX991V0
PX991WX
PX991WW
PX990RW
PX990NV
PX990QW
PX990WW
PX990WV
PX990WT
PX990WR
PX99020
PX9999R
0
-10
-20
-30
-40
-50
-60
BULK Batch
Cu
mu
lati
ve
Su
m
0UCL=3.79
LCL=-3.79
CUSUM Chart of % Label Claim
Process Complexity (Issue 4): Chronological Order
• Process data should be date-ordered a) Missing dates, b) Data Entry errors, c) Data not
sorted by date
• a) Missing dates Pattern of missing dates
• Missing At Random (MAR)
• Skipped due to delay in raw material testing results Lack of raw materials
• Skipped due to unplanned events such as inclement weather
• Missing Completely at Random (MCAR): e.g. SRS taken during investigations
• Missing Not At Random (NMAR)
• Intentional, e.g. Batch was scrapped
• Skipped due to sudden change in market demand for product change in
schedule
Process Complexity (Issue 4): Chronological Order
• Product X: Missing At Random (MAR), i.e. random sample of 40 without replacement
10810510299969390
LSL USL
LSL 90
Target *
USL 110
Sample Mean 98.367
Sample N 40
StDev (Within) 0.850155
StDev (O v erall) 1.06714
Process Data
C p 3.92
C PL 3.28
C PU 4.56
C pk 3.28
Pp 3.12
PPL 2.61
PPU 3.63
Ppk 2.61
C pm *
O v erall C apability
Potential (Within) C apability
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
O bserv ed Performance
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
Exp. Within Performance
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
Exp. O v erall Performance
Within
Overall
Process Capability of % Label Claim
Process Complexity (Issue 4): Chronological Order
• b) Data Entry Errors
• Dates entered incorrectly.
• c) Data not sorted by date
• Dates are recorded Process Data Not in chronological order
• Control Chart (Product X)
PX99WRW
PX994RW
PX99O0W
PX99N76
PX99PM0
PX991V0
PX99Q36
PX990RW
PX998MT
PX990NV
PX999NP
PX99O0X
PX992WN
PX99WRV
PX997VM
PX991V2
PX99020
PX991WW
PX99NQ9
PX990QW
PX9999R
PX997VN
PX99R9X
PX990WT
PX998T2
PX990WW
PX998MV
PX99Q35
PX998T0
PX99OV9
PX995WR
PX99OXM
PX999NR
PX998T1
PX9954M
PX991WX
PX99OV8
PX991V1
PX9996M
PX99RM8
PX990WR
PX99WR9
PX994RR
PX99676
PX99PXR
PX99677
PX997VP
PX993N1
PX99O7R
PX990WV
110
105
100
95
90
BULK Batch
Ind
ivid
ua
l V
alu
e
_X=98.49
UCL=102.90
LCL=94.07
110
90
I Chart of % Label Claim
Note: Data are not in chronological order
Process Complexity (Issue 4): Chronological Order
• Process Capability (Product X): Random shuffle
10810510299969390
LSL USL
LSL 90
Target *
USL 110
Sample Mean 98.4856
Sample N 50
StDev (Within) 1.47199
StDev (O v erall) 1.37822
Process Data
C p 2.26
C PL 1.92
C PU 2.61
C pk 1.92
Pp 2.42
PPL 2.05
PPU 2.78
Ppk 2.05
C pm *
O v erall C apability
Potential (Within) C apability
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
O bserv ed Performance
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
Exp. Within Performance
PPM < LSL 0.00
PPM > USL 0.00
PPM Total 0.00
Exp. O v erall Performance
Within
Overall
Process Capability of % Label Claim
In Chronological Order
Process Complexity (Issue 4): Chronological Order
-Suggested Solutions-
• Identify patterns/reasons for missing dates and recode where applicable.
• Increased awareness of missing date patterns
• Understand the distribution of missing dates
• Missing Date Imputation not an option
• Mitigation of missing dates via preventive or corrective actions
• Reduces compliance risk associated with inadequate disclosure
• Less likely to conclude that Process is Not Capable when it is Capable: Reduces Type
II Error
Process Complexity (Issue 5): Round-Off Errors
• Measurements are used to extrapolate to batch processe
• Quality of measurements degraded by round-off errors
• Round-off errors introduced during production
• Round-off errors introduced during data analysis
• Large round-off errors compromised data
Process Complexity (Issue 5): Round-Off Errors
• Intra-class Correlation ((ICC)
• Increase in measurement error Decrease in ICC
• Rule I Data value outside ±3SD
• Rule I; ICC = 0.5 88% chance of detection
Process Complexity (Issue 5): Round-Off Errors
-Suggested Solutions-
• In order for process behavior to be predictable uncertainty due to measurement error
needs to be minimized
• Minimize rounding during preliminary steps in data analysis
• When possible, round-off at the end of all calculations
• ICC can be used to quantify uncertainty due to round-off errors
• Impact of round-off error on prediction of process performance < other causes of
uncertainty.
Process Complexity(Issue 6): Random Fluctuations
• “Imprecise” lacking precision without being meaningless. Applicable when specification
limits cannot always be represented by precise numbers (Kaya and Kahraman 2008; Chen, Lai
et al. 2008; Parchami et al. 2014), e.g., Release limits were estimated from a process with
inherent random fluctuations.
• When Specifications are “Imprecise”, PCI is also “Imprecise”. Classic PCI do not apply.
• Example, specification limits are triangular numbers. This applies to certain quantitative quality
measurements, e.g. microelectronics manufacturing process and glass manufacturing.
Process Complexity (Issue 6): Random Fluctuations
-Suggested Solutions-
• Consider a PCI (FPCI), that is a generalization of the loss-based Taguchi PCI ( Hsu
and Shu 2008)
• Reduces to a classical test with a binary decision
• Computer a 100(1-α)% confidence interval for FPCI
• As increases, the tends to a sharper triangular number.
Process Complexity: Conclusions (1 of 2)
• Strategies for addressing common pitfalls in implementing Continued Process Verification
have been proposed:
• Small Sample Size
• Surrogate Process Capability Index based on Tolerance Intervals
• Can perform capability assessments for recently commercialized products
• Non-Normality
• Box-Cox transformation followed by ‘Normal’ Capability analysis
• Auto-correlation
• Sub-sampling if dataset is large and interval between batches is short.
• Split dataset if different batch sizes represented in the data
• Mean Shifts
• Address root-cause
Process Complexity: Conclusions (2 of 2)
• Chronological order
• Ensure that data are in chronological order
• Round-off Errors
• Carry out any intermediate calculations on the original scale
• Round-off final estimates
• Intra-class Correlation can be used to quantify uncertainty
• Random Fluctuations
• Loss-based Process Capability Index, Taguchi Index, is suggested
Forward-
Looking
Stateme
nts
This presentation may contain forward-looking statements
based on current assumptions and forecasts made by Bayer
Group or subgroup management.
Various known and unknown risks, uncertainties and other
factors could lead to material differences between the actual
future results, financial situation, development or
performance of the company and the estimates given here.
These factors include those discussed in Bayer’s public
reports which are available on the Bayer website at
www.bayer.com.
The company assumes no liability whatsoever to update
these forward-looking statements or to conform them to
future events or developments.