Posterior Probability of Passing Compendial Test

The Posterior Probability of Passing a Compendial Test (Pa)

Dave LeBlond, Principle Research Statistician, Abbott [email protected]

Linas Mockus, Research Scientist, Purdue University [email protected]

May 10, 2012 Bayes 2012, Aachen

2 Posterior Probability of Passing a Compendial Test Dave LeBlond May 10, 2012

Outline

Test Pass Probability, Pa

Bayesian PQ Approach

18th Century Statistics

+

Process Qualification (PQ)

ASTM E2709

Hierarchical process model

Summary 21st Century cGMPs

Prior Calibration Number of Batches for PQ?

Operating Characteristics Application to Sample Data

Cost Estimation 20th Century Computing

Compendial Tests

USP Dosage Uniformity

19th Century Regulations


What is a Compendial Test?

Bright line standard of quality. batch should always pass.

Bad Good

Fail Pass

Process Qualification ASTM E2709 USP905 Bayesian Approach Prior Calibration Number of Batches? Sample Data Summary

May use multi-stage sampling. USP & have 2 stages USP & have 3 stages

May use complex limits Indifference zones Limits on means, individuals, RSDs, counts Zero tolerance limits

Benchmark for setting batch acceptance criteria.


FDA Process Validation Guidance

Establishes 3 validation stages8 Process Design (QbD) Process Qualification (PQ) Continued Process Verification

[PQ] criteria [should] allow for a science- and risk-based decision about the ability of the process to consistently produce quality products

[and] include statistical metrics defining both intra-batch and inter-batch variability.8

Place acceptance limits on Pa, the probability that future batches will pass the compendial test.



Identify an acceptance region (AR) in parameter space: g( ) >= LB

Choose a 100(1-)% confidence region (CR) method, given a sampling plan and data.

ASTM E2709 Approach to PQ

Derive g: Pa >= g( )


Generate an acceptance table (AT): est such that CR is within AR.

Identify est .

Choose a sampling plan so that Prob( est is within AT) > some desired value.

Obtain data. If est is within AT, there is 100(1-)% confidence that Pa >= LB on repeated sampling of that batch.


Praise for ASTM E2709

This paradigm shift fundamentally changes how our industry should develop in-house specifications.2

[This new] concept of meeting specification may begin to include estimates of statistical confidence as part of cGMP.3

E2709 was highly effective in identifying nonconforming material.4 Available for USP & as CUDAL, a validated SAS

program,10 and as an Excel Spreadsheet.11

Clearly ASTM E2709 is a very positive step but ... Is anything missing?

Can Bayesian tools make further improvement?



ASTM E2709 Limitations

Parameters are fixed (cannot have a distribution). Pa is also denied a distribution. Prediction of failure rates for future lots requires integration

over the uncertain Pa Not allowed.

Confidence region approach Conservative approximations Biased predictions.


Confidence regions are not unique.

Qualifies 1 batch at a time. No inference about the process. What about cost?


USP Uniformity of Dosage Units: Quality

Stage Sample Size Requirements

0

2

4

6

8

70 100 130

Pass

Need stage 2 1

10

10SD

10X

2

20 more 0

2

4

6

8

70 100 130

Pass Fail 30SD

30X(+ individual limits met)



A Hierarchical Process Model

1 2 B Batches ( )2,~ Biid

i Nu Batch i mean potency:

Process Process mean potency:

Tablets ( )2,~ Ti

iid

ij uNy

Tablet j(i) observed:

[3]



Actual Production Data: Joint 95% CI

Each batch passed USP. B > 0 (Pvalue < 0.05)


B T

Product Batches LB est UB est UB est UB A 3 100.0 100.0 100.1 0.0 0.0 3.1 4.3 B 32 100.8 101.8 102.8 2.2 3.1 3.0 3.2 C 85 100.7 101.6 102.5 3.5 4.2 3.3 3.5 D 49 100.2 100.6 101.0 1.2 1.5 1.5 1.6 E 2 99.7 100.1 100.4 0.0 9.9 2.5 3.9 F 32 100.1 101.2 102.2 2.4 3.3 2.5 2.8 G 10 98.3 99.8 101.3 1.6 2.3 2.1 2.2 H 4 98.7 100.8 102.8 1.7 7.4 1.1 1.5 I 14 99.4 100.7 101.9 2.0 3.4 2.6 3.0 J 16 98.9 100.0 101.2 1.8 3.0 3.5 4.0 K 4 98.9 99.8 100.6 0.7 3.0 0.8 1.1 L 4 101.7 102.6 103.6 0.7 3.4 1.7 2.2


Pa is a deterministic function of model parameters Use Monte-Carlo simulation to generate lookup table.

T

2

4

6

95 100 105

0.10

0 0.100

0.200

0.200

0.30

0 0.300

0.40

0 0.400

0.50

0 0.500

0.60

0 0.6000.700

0.8000.900

0.9500.990

0.999

: B 295 100 105

0.100

0.100

0.20

0 0.200

0.30

0 0.300

0.40

0 0.400

0.500

0.5000.600

0.7000.800

0.9000.950

0.990

0.999

: B 3.5

95 100 105

0.200

0.200

0.30

0 0.300

0.400

0.400

0.5000.6000.7000.800

0.900

0.950

0.990

: B 5


Based on 10K simulated batches for each of 18K grid points. Use tri-linear interpolation to obtain Pa for any desired .


A Bayesian Approach to PQ

Generate (by simulation) an interpolation table for Pa = g( ) Choose PQ acceptance criteria (AC)

lower bound (LB) for Pa. require that 100(1-)% of the Pa posterior mass be >= LB.

Choose a sampling plan based on the simulated Operating Characteristics (OC).

Obtain data. Obtain a posterior sample of . Obtain a posterior sample of Pa.

If Pa posterior is acceptable, there is at least 100(1-)% probability that Pa >= LB for the process.

Mean of Pa posterior = expected probability that future batches will pass the compendial test.



Benefits of a Bayesian Approach to Qualification

Direct inference on the parameter of interest (Pa) Posterior distribution of Pa

Quantitative risk assessment (i.e., ICH Q9) Production planning (expected cost and throughput)

Leverage prior knowledge (if justified) [The qualification report should consider] the entire compilation

of knowledge and information gained from the design stage through the process qualification stage.8



Proposed Bayesian PQ Acceptance Criterion The confidence level selected can be based on risk analysis as it relates to the

particular attribute under examination.8

Pa

Per

cent

of T

otal

0

5

10

15

20

25

0.2 0.4 0.6 0.8

24 batches Median(Pa)=0.89

Pa

Per

cent

of T

otal

0

5

10

15

20

25

30

0.2 0.4 0.6 0.8

4 batches Median(Pa)=0.92


Propose: median(Pa posterior) 0.9 ( = 0.5, LB = 0.9) mean would be computationally simpler, more discriminating

Examples of simulated borderline cases = 105, B = 2, T = 4, 10 units/batch, Noninf Priors, 10K MCMC draws


Prior Calibration Using Simulated Data

B T Pa* Ps* 2 2 102 1.00 0.00 5 2 102 0.98 0.03 2 5 102 0.97 0.27 5 5 102 0.78 0.44 2 2 107 1.00 0.02 5 2 107 0.87 0.17 2 5 107 0.49 0.72 5 5 107 0.50 0.67

* based on 100K simulated batches each.

Simulated qualification data from 3, 4, 6, 12, or 24 batches 4 weakly informative priors examined


102 107

5

2

5

2

B

T


Weakly Informative, Independent Priors Used None Least More Most

70-130

N(100,10002) N(100,10002) N(100,152) N(100,102)

0-20

RIG(.001,.001) Undefined

mean df = 0

Half Cauchy* infinite mean

df = 1

Half t* mean = 70.7,

df = 2

Half t* mean = 14.1

df = 2

T 0-20


mean df = 0


mean df = 0

RIG(0.5,8) Mean = 4

df = 1


mean df = 0



% Bias in Pa Posterior Mean

All priors exhibit conservative (low) mean(Pa) Non-informative prior is least conservative.

Number of Batches

% B

ias

in P

a es

timat

e

-40-30-20-10

0

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

-40-30-20-100

: T 5 : B 5

: 102

Least More Most None

Number of Batches%

Bia

s in

Pa

estim

ate

-40-30-20-10

0

5 10 15 20

: T 2 : B 2

: 107

: T 5 : B 2

: 107

: T 2 : B 5

: 107

5 10 15 20

-40-30-20-100

: T 5 : B 5

: 107




% RMSE in Pa Posterior Mean

In most cases Non-informative prior has lower RMSE.

Number of Batches

% R

MS

E in

Pa

estim

ate

010203040

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

010203040

: T 5 : B 5

: 102


Number of Batches%

RM

SE

in P

a es

timat

e0

10203040

5 10 15 20

: T 2 : B 2

: 107

: T 5 : B 2

: 107

: T 2 : B 5

: 107

5 10 15 20

010203040

: T 5 : B 5

: 107




90% Credible Interval Coverage of Pa

Non-informative prior has coverage closest to nominal

Number of Batches

90%

Inte

rval

Cov

erag

e fo

r Pa

0.750.800.850.900.95

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0.750.800.850.900.95

: T 5 : B 5

: 102


Number of Batches90

% In

terv

al C

over

age

for P

a

0.750.800.850.900.95

5 10 15 20

: T 2 : B 2

: 107

: T 5 : B 2

: 107

: T 2 : B 5

: 107

5 10 15 20

0.750.800.850.900.95

: T 5 : B 5

: 107




Summary and Final Choice of Prior

% Bias in Pa Posterior mean Bias increases from -10% to 0% as #batches grows (3 to 24) In all cases, Non-informative prior shows least bias

% RMSE in Pa Posterior mean RMSE asymptotes from +10-30% to zero as #batches grows Non-informative prior shows least RMSE

Coverage of 90% Credible Interval Nominal coverage in most cases, some cases only 80% Non-informative prior coverage closest to nominal

Non-informative prior used here



Establishing a PQ Sampling Plan

Pa = quality metric

Poor Quality

Good Quality

Pa 0.4 1


RQL AQL How many batches (more/less) ?

Requires Monte-Carlo simulation

0.95

AQL

Acceptable Quality Level

0.10

RQL

Rejectable Quality Level Pro

b(Pa

ss P

Q) Operating Characteristic

Curve(OC)

Does OC depend on ?


OC of Proposed Approach

27 grid points 1000 data sets per grid point Vary number of batches

102 107 112

8

5

2

8 5

2

B

T


Number of Batches

Population Pa

(Pro

b(m

edia

n(P

a)

0.9))1

4

0

0.05

0.1

0.5

0.95

0.0 0.2 0.4 0.6 0.8 1.0

3 4 6 12 24

OC independent of ?


Using AQL and RQL to Set Number of Batches


Number of Batches

Population Pa

(Pro

b(m

edia

n(P

a)

0.9))1

4

0

0.05

0.1

0.5

0.95

0.0 0.2 0.4 0.6 0.8 1.0

3 4 6 12 24

0.10 0.95

AQL RQL


Popn.beta

102 107

Pop

n.si

g.T

2

00.20.40.60.8

1

Y 2

5

00.20.40.60.8

1

Y 5

Popn.sig.T

2 4 6 8 10 12Number of Batches


Popn.sig.B=5

Pa=0.98

Pa=0.87

Pa=0.78 Pa=0.50

= 5

PQ Pass Rates(=0.5, LB=0.90) : Bayes vs E2709


Popn.beta

102 107

Pop

n.si

g.T

2

00.20.40.60.8

1

Y 2

5

00.20.40.60.8

1

Y 5

Popn.sig.T



Popn.sig.B=2

Pa=1.00 Pa=1.00

Pa=0.97 Pa=0.49

Pro

babi

lity

of P

assi

ng P

Q

= 2


Actual Production Data

Note: Each lot passed USP.

Product C Product D

LotPotency

10 25 40

85100

115

Lot

Potency

10 25 40

85100

115



Joint Posterior Samples after 3, 6, and 20 batches Product C

T

B

Product D



0.05, 0.50 & 0.95th Posterior Quantiles for Product D

REML Point Estimates = 100.6 B = 1.2 T = 1.5

# of lots

Qua

ntile

s of

Pa

10 25 400

0.5

1 Cumulative Analysis (lots 3 to 45)


For very good process 3 may be enough.


0.05, 0.50 & 0.95th Posterior Quantiles for Product C

REML Point Estimates = 101.6 B = 3.5 T = 3.3

# of lots

Qua

ntile

s of

Pa

10 25 400

0.5

1 Cumulative Analysis (lots 3 to 45)


For challenging processes >3 required.


How Many Batches for a PQ?

2 approaches: Prospectively: use simulated OC curves, or Cumulative stability of Pa quantiles.



Cost Estimates

Not a regulatory consideration Important for Manufacturer

Stage testing increases analytical costs Failures risk supply of critical drugs Resource planning

Need Pa to estimate cost For multi-stage compendial tests, also need Ps, the probability

of stage testing.



USP Uniformity of Dosage Units: Cost Let Ct = Cost of testing 10 tablets ( ~ $3K ) Cm = Cost of manufacturing 1 batch ( ~ $200K ) Ci = Cost of a failure investigation ( ~ $2K )

Then E[total cost] = Ct + 2CtPs + Cm +Cm (1-Pa) + Ci (1-Pa)

Stage 2 triples testing cost Failure doubles manufacturing cost and requires a

failure investigation.



Posterior Distribution of Future Costs for Product C

Estimation after first 6 batches

Pa Posterior

Per

cent

of T

otal

0

20

40

60

0.6 0.7 0.8 0.9

Ps Posterior

Per

cent

of T

otal

0

10

20

30

40

50

60

0.1 0.2 0.3 0.4

Total Cost ($1K) Posterior

Per

cent

of T

otal

0

20

40

60

200 250 300 350

Pa Mean = 0.97 Ps Mean = 0.05 Total Cost Mean = $209K


Production costs per batch (posterior expectation): Testing: $3.3K Manufacturing: $206K Investigation: $60


Future Work

Compare Stringency of Proposed Bayesian Criterion with CUDAL.10, 11

Dependence of OC curve on ? Non-normal populations? Excel tool? Extension to any compendial test

Same principles Most will be trivial extensions, compatible with Excel

Dissolution & disintegration multi-stage tests USP, ,



Conclusions

Pa, is a key quality metric in PQ. Ps may also be of interest to manufacturers. ASTM E2709 is a breakthrough in PQ thinking. PQ requires a model for between-batch variance. Bayesian hierarchical modeling provides

Direct inference on Pa and Ps Basis for sampling plan choice Manufacturing cost projections



References 1. American Society for Testing and Materials (ASTM) Standard E11 2709-10, May, 2010 2. Torbeck, LD (May 2, 2010) Statistical solutions: Bergums method recognized, Pharmaceutical

Technology. 3. Jon Clark (September 29, 2010) Confidence- critical to batch release: Application of ASTM E2709,

presented at QbD/PAT Conference, University of Heidelberg. 4. Lunney, P.D., Anderson, C.A., Investigation of the Statistical Power of the Content Uniformity Tests

Using Simulation Studies, Journal of Pharmaceutical Innovation, pp 24-35, 13March2009. 5. US Pharmacopoeia 34 (2011) General Chapter Uniformity of Dosage Units (harmonized with JP

and EP). 6. LeBlond, DJ (Spring, 2005) Methodology for predicting batch manufacturing risk. MS Thesis, Colorado

State University. 7. LeBlond DJ (August, 2009) Risk Assessment of Drug Product Content Uniformity Release Failure: A

Bayesian Approach, Joint Statistical Meetings, Washington DC 8. FDA CDER, CBER, CVM (January 2011) Guidance for Industry, Process Valdiation: General Principles

and Practices, rev 1. 9. Gelman, A (2006) Prior distributions for variance parameters in hierarchical models, Bayesian Analysis

1(3), 515-533 10. J.S. Bergum and L. Hua (October 2, 2007), Acceptance Limits for the New ICH USP 29 Content-

Uniformity Test, Pharm. Technol. Online http://pharmtech.findpharma.com/pharmtech/article/articleDetail.jsp?id=463577), accessed Apr. 4, 2012, October 2007.

11. P. Cholayudth (2009), Establishing Acceptance Limits for Probability of Passing Multiple Stage Tests in Proces Validation through a Process Capability Approach, Jrnl. of Validat. Technol. 15 (4), 7790.

12. Y. Hu and D. LeBlond (2011) Assessment of Large-Sample Unit-Dose Uniformity Tests, Pharmaceutical Technology 35(10) 82-92.


Backup Slides


Full Conditionals for Gelmans Half-t Prior9

m ( )

2

2 2 11

1| | ,2 2

i

B

iBp IG

=

+ + =

( )

( )

2 2

12

1 1

| , , , | ,2 2

i

T T T T

B

ii

TB

ij ii j

i i

N Sp IG a b

where

N T

S y u

u

=

= =

= + +

=

=

= +

y

( )( ) 2

1 122 2

2 22 2

1 1

| , , , | ,

iTB

i iji j T

T B BT T

i i i ii i

yp N

T TA A

= =

= =

=

+ +

y

( )( )2 2 2

2 2 12 2 2 2 2 2| , , , , | ,

iT

ijTi

i T iT i T i

yp N

T T

=

=+ +

y

( )( )2 20 0 2 2

1 12 02 2 2 20 0

| , , , | ,

iTB

T ij ii j T

TT T

yp N

N N

= =

+

= + +

y

2 2 2B =


%Bias in posterior mean

Non-informative prior is least biased at lower number of batches Bias may be induced by prior mean

Number of Batches

% B

ias

in ^

-1.0

-0.5

0.0

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

-1.0

-0.5

0.0

: T 5 : B 5

: 102


Number of Batches%

Bia

s in

^

-1.0

-0.5

0.0

5 10 15 20

: T 2 : B 2

: 107

: T 5 : B 2

: 107

: T 2 : B 5

: 107

5 10 15 20

-1.0

-0.5

0.0

: T 5 : B 5

: 107



%RMSE in posterior mean

Virtually identical for =107 Non-informative prior has lowest loss based on MSE

Number of Batches

% R

MS

E in

^

1

2

3

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

1

2

3

: T 5 : B 5

: 102



90% Interval Coverage of beta

Coverage is nearly nominal for Non-informative prior regardless of number of batches.

Number of Batches

90%

Inte

rval

Cov

erag

e fo

r

0.85

0.90

0.95

1.00

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0.85

0.90

0.95

1.00 : T 5 : B 5

: 102



% Bias in sB and sT Posterior Means

Virtually identical for beta = 107 Non-informative prior has lowest bias. Bias may be induced by prior

mean.

Number of Batches

% B

ias

in ^

B

0100200300400500

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0100200300400500

: T 5 : B 5

: 102


Number of Batches%

Bia

s in

^T

0

2

4

6

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0

24

6

: T 5 : B 5

: 102



% RMSE in sB and sT Posterior Means

Virtually identical for beta = 107 For sigB, Non-informative prior has least RMSE. For sigT, prior choice

is irrelevant.

Number of Batches

% R

MS

E in

^B

0

200

400

600

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0

200

400

600 : T 5 : B 5

: 102


Number of Batches%

RM

SE

in ^

T

68

101214

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

68101214

: T 5 : B 5

: 102



Coverage of 90% interval estimates of sB and sT

Virtually identical for beta = 107 For sigB, Non-informative prior coverage is a little low in some cases.

For sigT, prior has little effect on coverage.

Number of Batches

90%

Inte

rval

Cov

erag

e fo

r B

0.80

0.85

0.90

0.95

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0.80

0.85

0.90

0.95

: T 5 : B 5

: 102


Number of Batches90

% In

terv

al C

over

age

for

T

0.880.890.900.910.92

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0.880.890.900.910.92

: T 5 : B 5

: 102



% Bias and % RMSE in the Ps Posterior Mean

The % bias for 102, 2, 2 was extremely high because the population Ps is so close to zero. The % RMSE is nearly identical because the bias is the largest contributor to MSE Non-informative prior has lowest % Bias

Number of Batches

% B

ias

in P

s es

timat

e

050

100150200250

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

050100150200250

: T 5 : B 5

: 102


Number of Batches%

Bia

s in

Ps

estim

ate

050

100150200250

5 10 15 20

: T 2 : B 2

: 107

: T 5 : B 2

: 107

: T 2 : B 5

: 107

5 10 15 20

050100150200250

: T 5 : B 5

: 107



Coverage of the 90% Credible Interval for Ps

Non-informative prior has coverage closest to nominal Number of Batches

90%

Inte

rval

Cov

erag

e fo

r Ps

0.40.50.60.70.80.9

5 10 15 20

: T 2 : B 2

: 102

: T 5 : B 2

: 102

: T 2 : B 5

: 102

5 10 15 20

0.40.50.60.70.80.9

: T 5 : B 5

: 102


Number of Batches90

% In

terv

al C

over

age

for P

s0.40.50.60.70.80.9

5 10 15 20

: T 2 : B 2

: 107

: T 5 : B 2

: 107

: T 2 : B 5

: 107

5 10 15 20

0.40.50.60.70.80.9

: T 5 : B 5

: 107



OC of Proposed Approach: Effect of model parameter (one at a time, Non-informative prior)

Population parameters varied beta: 100-120 with

sigT=sigB=3 sigB: 2-15 with sigT=2 and

beta=102 sigT: 2-15 with sigB=2 and

beta=102

In principle, the median(Pa) and its variability may depend on: Population Pa Population Parameters.

This graph shows that model parameter has a minor affect on the OC curve, the OC curve is controlled largely by the population Pa. Population Pa

Pro

b[m

edia

n(P

a) >

= 0.

9]

0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.2 0.4 0.6 0.8 1.0

: B 3

VariedbetasigB

sigT


Operating Characteristics of Proposed Approach: Effect of prior, number of batches, choice of statistic

Non-Informative Prior least sensitive to sample size Mean(Pa) is more conservative than median(Pa)

Population Pa

Pro

b[m

edia

n(P

oste

rior P

a)>0

.9]

0.0

0.2

0.4

0.6

0.8

1.0

0.5 0.6 0.7 0.8 0.9 1.0

: Prior Least : Prior More

: Prior Most

0.5 0.6 0.7 0.8 0.9 1.0

0.0

0.2

0.4

0.6

0.8

1.0 : Prior None

Number of Batches34

612

24

Population PaP

rob[

mea

n(P

oste

rior P

a)>0

.9]

0.0

0.2

0.4

0.6

0.8

1.0

0.5 0.6 0.7 0.8 0.9 1.0

: Prior Least : Prior More

: Prior Most

0.5 0.6 0.7 0.8 0.9 1.0

0.0

0.2

0.4

0.6

0.8

1.0 : Prior None

Number of Batches34

612

24

Posterior Probability of Passing Compendial Test

Documents

Transcript of Posterior Probability of Passing Compendial Test