Pierre Lebrun | Director statisticsBruno Boulanger | CSOTimothy Mutsvari | Manager statistics

CASSS > THE NETHERLANDS AREA BIOTECH (NLab) DISCUSSION GROUPNijmegenSeptember 2018

Investigating Statistical Methodology for the Comparison ofQuality Attributes in Drug Development

EMA reflection paper

Statistical methodologies

Feel the value of Bayesian statistics

Conclusions

EMA reflection paper:

Unified set of recommendations about statistical methodologies for three different questions:

Comparability of processes after a change Small LargeBiosimilar product ------ LargeGeneric product Small -----

What else ? To answer these questions, comparability of the measurement system should be included in the story…− Analytical transfer, bridging, etc.− Is the Measurement System good enough to follow the (reference and test) processes ?− Often, process changes come along with assay updates

What makes them different or common?

Large moleculeShort history− New assays− Many assays− No clinical data available

ProcessSpecifications unknown

Small moleculeLong history− Same assays− Few assays− Clinical data available

ProductSpecifications known

Small molecules vs. large molecules

For small molecules, there are unambiguous ways to assess they are structurally the same. Consistency and adequacy of the formulation process is then central.For large molecules, there is no unambiguous ways to assess the products are the same

− The reason the word “similarity” is usedFor large molecules, subtle changes in process may have important consequencesThe number of Quality Attributes

− Small for small molecules− Large for large molecules, often highly correlated

Long history vs limited history

In pre/post manufacturing changes

− Reduced list of Critical Quality Attributes

− Many “reference” batches available and few “test” batches usually envisaged

− Number of “test” batches not really a matter of debate

In Biosimilars

− Large list of Quality Attributes

− Several “reference” and several “test” batches are required

− Sample size computation to reach acceptable probability of success is a matter of debate

When Biosimilar company evaluates Reference products

− not always sure about the independency of batches (e.g. 3 in a short timeframe), age, etc.

Same assays or new assays

In pre/post manufacturing changes − The assays used are the same and therefore consistency of results is assumed− Assay variance impact on the process variance is “known”− The format of the reportable results is (should) be appropriate

In biosimilars − New assays need to be developed and validated− Similar (as possible) assays should be envisaged− What is the contribution of assay component and format ?

For generics− Mostly physico-chemical procedures whose overall performances are less an issue

Many QAs or limited number of QAs

In pre/post manufacturing changes− the number of Quality Attributes to be evaluated is reduced− The list is less prone to debate given history− The multiplicity issue is limited

In Biosimilars− The number of Quality Attributes to be evaluated is large− The list is a matter of debate and agreement− The multiplicity issue is rather important

Generics− Same as in pre/post manufacturing changes

Specifications and acceptance limits

In pre/post manufacturing change− the specifications are known and constant values.

For biosimilars− specifications are (by definition) unknown

• Originator have limits that are linked to clinical relevance− should be established and justified and therefore are random variables.

In pre/post manufacturing change− specifications are about individual batches (individual CoA).− Why should it be different for biosimilars

How to map from specifications on individual batches to acceptance limits on parameters such as mean?For small molecules, there are already several “good practices” fixed limits defined (“80%-125%” rule, CU, 98%-102%, ….)

Specifications: Clinical data available

If a “reference” product is on the market − it is within specifications− It is clinically acceptable

The range of values obtained for “reference” batches − Are by definition clinically acceptable values and justified− Do applies to the individual batches and are natural “acceptance limits”− How to figure out the real range of values patients are exposed to ?

How can “acceptance limits” be built for the mean or variance based on range of individual batches ? − The mean and variance precision of estimates depends on the number of data− Another arbitrary constant such as 1.5 (as in TOST) or 1/8 should then be invoked for biosimilars

• Might be too small, or too large− For generics, there are already criteria established since a long time (eg 80%125%) that have a

proven relevance

EMA reflection paper

Statistical methodologies

Feel the value of Bayesian statistics

Conclusions

Methodologies for statistical equivalence

Observational / data and graphical comparisonSimple t-testEquivalence test (TOST)“Quality Range Approach”Tolerance intervals / Probability to meet the objectives

12

”A huge diversity of comparison approaches has been applied in the past, and some of them included statistical intervals, e.g. tolerance intervals, x-sigma, min-max range interval, etc. From a statistical perspective, the context of use of these intervals is however rarely clear in relation to the interpretation of conclusions drawn”

t-test (never use it!)

“Also, the lack of significant differences alone does not imply similarity. Hence, the question of whether the desired conclusion of similarity of products could indeed be inferred from often limited information from sample data remains difficult to answer in many occasions.”

13

0

0

Similarity rejected

Similarity accepted

!" − $ %&','&)*+, ;

!" + $ %&','&)*+,

P-values and CI

14

3.75 5

Null hypothesis not rejectedDoes not imply that the mean is 3.75 !!

Null hypothesis rejectedOkey, mean is (probably) not 5

Under H0: mean of the originator ?

Effect of Sample Size

An increase of the sample size leads us to reject H0 that the mean is equal to 3.75This hypothesis was previously not rejectedWhen we do not reject H0 it is often a problem of small sample size. It does not mean that we accept H0t-test not answering our questionTo show equivalence, we need equivalence margins !

15

3.75

Other Decision Procedures – FDA Tier Approach (withdrawn guidance)

Tier 1 – Most Critical Quality Attributes(1-2α)100% two-sided Confidence Interval for Difference in Means contained within +/-1.5!"#$

Tier 2 – Moderate Critical Quality AttributesQuality Range Method: mean +/- k !"#$

Tier 3 – Least Critical Quality AttributesRaw Data/Graphical Comparison

Compares the means of the two distributions - TOST

Compares the central portions of the two distributions

No ‘formal’ assessment of the two distributions

If the 90% CI on the difference µ1 - µ2 is included in [-Δ; Δ], it means that the test

succeeded with 95% confidence

Nb: confidence level is corrected for double testing (multiplicity)

a = 0.1 à 95% confidence

-Δ Δ

90% CI on µ1 - µ2

H0 : µ1 - µ2 ³ D HA : µ1 - µ2 < DH0 : µ1 - µ2 ≤ -D HA : µ1 - µ2 > -D

Two-One Sided Tests (TOST)

Now, if the sample size increases, there is more chance to detect a similarity (up to Δ)

If this similarity is really there… at least it goes in the right direction

!" − $ %&','&)*+, ;

!" + $ %&','&)*+,

0

The problem of the TOST

“Making use of inferential statistical methods means quantifying uncertainties arising from the fact that claims (or decisions) are made based on limited data stemming from a sample. If comparative data analysis is limited to the sole description of the samples taken, it is evident that no clear inference can be drawn regarding drug material that was not sampled.”“Therefore, the understanding that interest is not so much in the actual samples (e.g. batches) drawn, but in the underlying (actually unknown) data distributions of the entirety of the materials ever produced, is key to follow the considerations in this document.”

That is to say: if you only look at the population parameters (e.g. the mean, the variance), you are looking at the data at hands, collected YESTERDAY. But the real question is to know if you WILL be able to reproduce your biosimilar TOMORROW. And this is where the real -predictive- statistical inference comes in handy !

What do we mean by similar/comparable?

Demonstrate that proposed new process produces lots of Test product that are (analytically) “equivalent/comparable” to those of the Reference product (both now and in the future).

When are the two distributions equivalent/comparable?

ü These distributions are the predictive distributions of the observed dataü They explain where FUTURE data would be found

Process or Product ?

When dealing with CMC and Quality Attributes:Should the “acceptance limits” apply − to the Process and individual units ?− to the Product and the means and/or the variances?

How to justify clinically defendable limits for mean or variance of process? The decision should be made on current (past) batches, or on future “capability” to produce lots within “acceptance limits” given observations.The range of the batches is important for the patient safety and efficacy.

What do we mean by similar/comparable?

Demonstrate that proposed new process produces lots of Test product that are (analytically) “equivalent/comparable” to those of the Reference product (both now and in the future).

When are the two distributions equivalent/comparable?

What do we mean by similar/comparable?

Demonstrate that proposed new process produces lots of Test product that are (analytically) “equivalent/comparable” to those of the Reference product (both now and in the future).

When are the two distributions equivalent/comparable?

A Definition of Biosimilarity

The test product is analytically comparable (for a given attribute) to the reference product if the middle P% of all lots produced by the Test product process lie within the middle P% of the lots produced by the Reference product process.Example, P = 99%.

P

O P

OIn the absence of

specification

limits

Combinations of Mean and SD that would be considered Biosimilar

A Definition of Biosimilarity

A Decision Procedure for Biosimilarity (1: the Reference)

Interested in limits defined by central portion of distribution of Reference product lotsMean and variance of Reference estimated with uncertaintyThe b-content g-Confidence Tolerance Interval (TI) on Reference is recommended

Takes into account the uncertainty on the Mean and the VarianceBetter statistical properties than Min and MaxBoth Content and Confidence can be controlled

Sufficient testing of the Reference is recommended to make b-content g-Confidence Tolerance Interval (TI) relevant for Similarity limits. (Here we will use N=15)− Otherwise the defined limits will be overly large and it is too easy to meet them with bad Test

product !!

( ){ } gb =>+<<- RefRefRefRefRefsXskXXskXPPk RefcRefcRefXsXc ,: ,

RefskX cRef ´±Tables and

software for

obtaining

kc are readily available

• Example: use 99%-Content 95%-Confidence Tolerance Intervals

Test if b-expecation Interval (Prediction Interval) of biosimilar (test sample) is within b-g-Tolerance Interval (TI) of reference

More relevant than using an arbitrary c factor (such as 3!)Takes into account the variability of the Test process (e.g. between-lots)Takes into account uncertainty on means and variability of Test processDemonstrates that a proportion b of the Test lots is expected to be within the range of Reference lots− That’s about the future

A Decision Procedure for Biosimilarity (2: the Test)

( ) ( ) TestTestnTest nstXTest

/111,2/1 +´± -+b

A Decision Procedure for Biosimilarity (Visualisation)

Test if b-Prediction Interval is within b-g-Tolerance Interval

This works

preferably well if

the ‘test’ is less

variable than the

reference !Rewards for good

manufacturing !

Acceptance limits

Better is… well…. better

(…) this does not necessarily mean that QAs of the pre-change and the post-change product are identical, but that the goal is to show that they are 'highly similar' in a sense that marked differences which would have adverse impact upon safety and efficacy of the drug product can be ruled out. Further interpretation of this wording might also indicate that manufacturing change-triggered differences in product quality, which are associated with positive impact on safety and/or efficacy, could eventually be accepted from the regulatory perspective

Probability of success

Same idea than for interval-based decision, but would rather directly compare the “test” data to the registered (generic) or determined (biosimilar) specifications

Specification limits

Predictive distribution

e.g. There is 95% guarantees that the testproduct will be of satisfactory quality

Predictive distribution

Reality is often complexIn many experiments, not only one source of variability (e) is of interest

e.g. here: the repeatability, the between analytical run and batch variances

Titer (log 10 IU/ml) vs. Unique batch identification (product;batch;month;condition)

Batch1 Batch2 Batch3 Batch4 Batch5

Tite

r (lo

g 10

IU/m

l)

9.3

9.4

9.5

9.6

9.7

9.8

9.9

10.0

10.1Run

123

Between batchesBetween runs (inside a batch)

look how homogenous !Between replicates (inside run and batch)

look how homogenous !

à The definition of intervals becomesmore intricate in this case

Total

EMA reflection paper

Statistical methodologies

Feel the value of Bayesian statistics

Conclusions

The objective: is my process biosimilar?

How to make a decision ?

A

B

What is the probability of obtaining the observed data, if the process is not biosimilar?

What is the probability being biosimilar, given the observed data?

2 / 18

Two different ways to make a decision based on

APr #$%&'(&) )*+* ,#+ $-#%-.-/*' )

n Better known as the p-value concept

n Used in the null hypothesis test (or decision)

n This is the likelihood of the data assuming an hypothetical explanation (e.g. the “null hypothesis”)

n Classical statistics perspective (Frequentist)

BPr $-#%-.-/*' #$%&'(&) )*+* )

n Bayesian perspective

n It is the probability of efficacy giventhe data

The Bayesian

perspective allows

to directly address

the question of

interest.

3 / 18

What is the question: − what is the probability of being biosimilar given available data?− what is the probability of having future lots within the limits given available data?

The question becomes naturally BayesianMany decisions can be deduced from the posterior and predictive distributionsIn addition− leverage historical data (e.g. on assay variability)− Bayesian approach can easily handle multivariate problems

Why Bayesian for Biosimilarity?

Pr $%&%'( )*&+ ,- ),.,&+ */+('0(1 12&2)vsPr */+('0(1 12&2 -*& /,*+,.,)2') Pr 4,*+,.,)2' */+('0(1 12&2)O P

35

Last points about Bayesian statistics− The interpretation is more natural− The learning curve may feel very steep (likely the reason why it is barely taught in

schools and universities)− It is not steep, but lack of simple, visual, practical resources to learn it− Once you have it, you can solve nearly all statistical questions using one single very

simple and beautiful framework• Relying on Monte-Carlo simulations, so you never get trapped into pages of complex

mathematics… you let the computer does what it does the best : compute• Even getting a monster like the “predictive distribution” is done quickly and easily

− It is finally possible to incorporate prior knowledge, because who has none of it?

Why Bayesian for Biosimilarity?

© PharmaLex

EMA reflection paper

Statistical methodologies

Feel the value of Bayesian statistics

Conclusions

Sampling plan

“Considerations concerning the sampling strategy are of utmost importance, and are expected to include the decision what the unit of observation will be: batches, lots, packages, tablets, vials/pools of liquid formulations, powders, etc. Decisions in this regard shall also be driven by the knowledge on potential sources of variability in the QA data”

Strong parallelism to Process Performance Qualification (PPQ) from Process validation FDA guidance

Sample size determination, i.e. justification of the choice of the many batches (not 3), locations (not 3), samples (not 3!), formulations (…), analytical sampling (runs and replicates) etc. should be taken most seriously !− E.g. If this is demonstrated that an assay does not suffer from between run variability, it is possible

to exclude this variable from the sampling plan and hence simplify data collection

Statistical simulations is the proven way to get a good rationale for the sample size.

© PharmaLex 37

Conclusions

Pre/post change manufacturing

Specifications knownReduced list of QAs –limited multiplicity issuesMany “reference” and few “test” batchesSame assays with overall contribution appropriate

Acceptance limits available

Biosimilars

Specifications unknownlarge list of QAs – non-ignorable multiplicity issueMany “reference” and many “test” batchesNew assays with overall contribution unknown

Acceptance limits to justify

Generics

Specifications definedReduced list of QAs–limited multiplicity issuesFew “reference” and few “test” batchesPhysico-chemical assays with overall contribution appropriate

Acceptance limits available

Is it possible to find a unified statistical methodology given those differences ?Wouldn’t it be better to create 3 guidance documents to avoid confusion ?

Three Fundamental Requirements

Define what you mean by equivalence/comparability/similarityProvide a well-defined decision procedure− Sample sizes− Metrics (intervals, probability, observational, etc.)− Experimental unit− …

Demonstrate the operating characteristics of the procedure (not much covered today)− What is the probability of deciding in favour of equivalence/comparability?− What is the patient risk?

• (Test product is deemed equivalent/comparable and a patient receives a bad lot from the Test product)− What is the producer risk?

• (Test product is deemed not to be equivalent/comparable when it is)

… and measurement systems…

THANKS !

Pierre LebrunDirector Statistics

PharmaLex Belgium SARue E. Belin, 5 1435 Mont-St-Guibert, Belgiumpierre.lebrun -at- pharmalex.com