Equivalence, Similarity, and Non-inferiorityEquivalence, Similarity, and Non-inferiority

Clinical Trials in NeurotherapeuticsClinical Trials in NeurotherapeuticsASENT 10ASENT 10thth Annual Meeting Annual Meeting March 2008March 2008

Marc K. Walton, M.D., Ph.D.Marc K. Walton, M.D., Ph.D.Senior Medical Policy AdvisorSenior Medical Policy AdvisorOffice of PolicyOffice of PolicyOffice of the Commissioner, FDAOffice of the Commissioner, FDA

The views expressed are those of the author, and do not represent an official FDA position

Equivalence / Similarity / NoninferiorityEquivalence / Similarity / Noninferiority

Distinctions of study circumstances and study goalDistinctions of study circumstances and study goal Need to be clearly identifiedNeed to be clearly identified

Two treatmentsTwo treatments Both known efficaciousBoth known efficacious Want to know comparative efficacyWant to know comparative efficacy

One treatment with unknown efficacyOne treatment with unknown efficacy Second treatment known efficaciousSecond treatment known efficacious Want to prove first is also efficaciousWant to prove first is also efficacious

Comparative Efficacy Comparative Efficacy

Can the two treatments be shown Equivalent?Can the two treatments be shown Equivalent?

Are two agents are essentially the same?Are two agents are essentially the same? Implies same amount of efficacy for each treatment to Implies same amount of efficacy for each treatment to

within some level of disinterest within some level of disinterest Neither drug is better than the otherNeither drug is better than the other

Two sided interestTwo sided interest Not a regulatory requirement for marketing Not a regulatory requirement for marketing Not common regulatory interest (ie, sponsor interest, Not common regulatory interest (ie, sponsor interest,

possibly with desire for regulatory affirmation)possibly with desire for regulatory affirmation)

““The Same”?The Same”? With respect to characteristics planned for rigorous With respect to characteristics planned for rigorous

evaluationevaluation

Proof of Efficacy for an Unproven TreatmentProof of Efficacy for an Unproven Treatment

Is the new treatment efficacious?Is the new treatment efficacious? In the same manner as the established treatmentIn the same manner as the established treatment

Often a goal for regulatory decision Often a goal for regulatory decision Based on evidence that new drug has Based on evidence that new drug has somesome efficacy efficacy No explicit requirement for same efficacy of other No explicit requirement for same efficacy of other

drugs, or any predefined fraction of efficacy of other drugs, or any predefined fraction of efficacy of other drugs for same disorder. drugs for same disorder.

One sided comparison of interestOne sided comparison of interest Precise comparative efficacy is not a goalPrecise comparative efficacy is not a goal

Non-inferiority: A misnomer Non-inferiority: A misnomer Goal to is to show “sufficient efficacy” Goal to is to show “sufficient efficacy” Accomplished by showing “not unacceptably inferior” Accomplished by showing “not unacceptably inferior”

to controlto control Does not imply truly not inferiorDoes not imply truly not inferior

SimilaritySimilarity

Perhaps not equivalent, but close enoughPerhaps not equivalent, but close enough ‘‘Close enough’?Close enough’?

In the eye of beholderIn the eye of beholder May imply a two sided interest as wellMay imply a two sided interest as well Individual judge dependentIndividual judge dependent

Difficult to define/describeDifficult to define/describe Principles of Equivalence / Noninferiority studies applyPrinciples of Equivalence / Noninferiority studies apply

More laxity in interpretation granted on an individual More laxity in interpretation granted on an individual person basisperson basis

Equivalency & Noninferiority: Active Control StudiesEquivalency & Noninferiority: Active Control Studies

Why / When do Active Control studies?Why / When do Active Control studies? Comparison of effects of two agentsComparison of effects of two agents

May not need to exclude PlaceboMay not need to exclude Placebo Issues of assay sensitivity (later) and validity of Issues of assay sensitivity (later) and validity of

interpretation much easier if placebo includedinterpretation much easier if placebo included Assessment of effects of one agent when placebo Assessment of effects of one agent when placebo

control not permissiblecontrol not permissibleEarlier lecture touched on circumstancesEarlier lecture touched on circumstancesEthics of withholding a known effective treatment Ethics of withholding a known effective treatment

with life-saving or irreversible life-altering effectwith life-saving or irreversible life-altering effectCan be dose-ranging study with single agentCan be dose-ranging study with single agent

Ethical need to avoid non-effective dosesEthical need to avoid non-effective doses May not be a practicable approach May not be a practicable approach

Remainder of talk assumes active control study of two drugsRemainder of talk assumes active control study of two drugs

Active Control StudiesActive Control Studies

Study aspects enabling data to be validly interpretedStudy aspects enabling data to be validly interpreted Validity of DesignValidity of Design Integrity of ConductIntegrity of Conduct

Adherence to protocolAdherence to protocol

Same overall issues with standard placebo-control Same overall issues with standard placebo-control studiesstudies Additional complexities in Active Control studiesAdditional complexities in Active Control studies

Increased ComplexitiesIncreased Complexities

Assay sensitivity of the designAssay sensitivity of the design Interpretation of the quantitative resultInterpretation of the quantitative result

Determination of margin of acceptable differenceDetermination of margin of acceptable difference

MarginMargin Built on combination ofBuilt on combination of

Historical knowledge of comparator’s efficacyHistorical knowledge of comparator’s efficacyClinical judgment of what is an ‘acceptable’ Clinical judgment of what is an ‘acceptable’

differencedifference

Assay SensitivityAssay Sensitivity

Can study detect a difference if one exists?Can study detect a difference if one exists? Unrecognized failure of assay sensitivity leads to:Unrecognized failure of assay sensitivity leads to:

Type II error for superiority studyType II error for superiority study Type I error for non-inferiority studyType I error for non-inferiority study Factors which affect sensitivity can seem to impair or Factors which affect sensitivity can seem to impair or

advance study organizer’s goals oppositelyadvance study organizer’s goals oppositely

Promotes ability to distinguish between Promotes ability to distinguish between Evidence of Absence Evidence of Absence Absence of EvidenceAbsence of Evidence

Laboratory assays often include positive and negative Laboratory assays often include positive and negative controls; rarely can be done in clinical trialscontrols; rarely can be done in clinical trials

Interpretation of Study ResultsInterpretation of Study Results

Results analyzed as comparison of two groupsResults analyzed as comparison of two groups Need numeric criterion to form interpretationNeed numeric criterion to form interpretation

Placebo control & other superiority studies have same Placebo control & other superiority studies have same requirement, but easy to definerequirement, but easy to define Show between-group difference > 0Show between-group difference > 0 New drug superior to placebo / other controlNew drug superior to placebo / other control

Superiority of new treatment over old treatment difficult to Superiority of new treatment over old treatment difficult to achieve with active agentsachieve with active agents Thus non-inferiority approach attractiveThus non-inferiority approach attractive Need to have quantitative comparison criterion to Need to have quantitative comparison criterion to

interpret study result, allowing for potentially the same interpret study result, allowing for potentially the same efficacyefficacy

Margin of AcceptabilityMargin of Acceptability

How much less efficacious can the new drug be than old How much less efficacious can the new drug be than old drug, and still deem it acceptable to use (or equivalent)?drug, and still deem it acceptable to use (or equivalent)?

Two components to considerTwo components to consider First – what is the efficacy the old drug provides?First – what is the efficacy the old drug provides?

Statistical analysis of existing informationStatistical analysis of existing informationCan not allow new drug to be worse than old drug Can not allow new drug to be worse than old drug

by that amount, or it provides no efficacyby that amount, or it provides no efficacy Second – how much of that quantitative amount of Second – how much of that quantitative amount of

efficacy is it permissible to give up?efficacy is it permissible to give up?Clinical judgmentClinical judgmentUsually easier to assess after know first Usually easier to assess after know first

componentcomponentExpress as an absolute amount or relative amount Express as an absolute amount or relative amount

Margin of AcceptabilityMargin of Acceptability

Two components – M1, M2Two components – M1, M2 Separate basis, separate purposeSeparate basis, separate purpose

Margin of acceptable inferiority cannot be larger than:Margin of acceptable inferiority cannot be larger than: Statistical component – M1Statistical component – M1 Clinical component – M2Clinical component – M2

Overall margin (M)Overall margin (M) If M2 << M1: M = M2If M2 << M1: M = M2 M2 > M1: Either no need for AC study (use placebo M2 > M1: Either no need for AC study (use placebo

control study) or non-inferiority not feasiblecontrol study) or non-inferiority not feasible M2 <M2 <(modest)(modest) M1 leads to complexities and anxieties M1 leads to complexities and anxieties

Active Control Agent Existing KnowledgeActive Control Agent Existing Knowledge

Solid knowledge of efficacy of active control agent is essentialSolid knowledge of efficacy of active control agent is essential From prior studies – historical placebo control studiesFrom prior studies – historical placebo control studies Preferably from multiple different prior studiesPreferably from multiple different prior studies

Historical studies each conducted in a specific mannerHistorical studies each conducted in a specific manner Regimen of use, population of useRegimen of use, population of use Concomitant care, alternative therapiesConcomitant care, alternative therapies Existing knowledge applicable in circumstances of prior Existing knowledge applicable in circumstances of prior

experienceexperienceMay not be reliable in other circumstancesMay not be reliable in other circumstances

Quality of historical studies needs to be consideredQuality of historical studies needs to be considered Adherence to protocolAdherence to protocol Quality of data collection for outcome now of interest (may Quality of data collection for outcome now of interest (may

not have been primary outcome in study design)not have been primary outcome in study design)

M1 – Statistical Margin ComponentM1 – Statistical Margin Component

What is the treatment effect of the comparator agent?What is the treatment effect of the comparator agent? Typical?Typical? Reasonably likely present?Reasonably likely present? Highly likely present?Highly likely present? There will be no ability to actually confirm in new AC There will be no ability to actually confirm in new AC

trialtrial Generally derived from some form of meta-analysisGenerally derived from some form of meta-analysis

With allowance for uncertainty in historical With allowance for uncertainty in historical quantitative estimates (i.e., variance of each effect quantitative estimates (i.e., variance of each effect estimate; variance of meta-estimate)estimate; variance of meta-estimate)

The more precisely treatment effect was estimated in The more precisely treatment effect was estimated in historical trials and the more trials there were, the historical trials and the more trials there were, the more precise the M1 value may be determinedmore precise the M1 value may be determined

M1 – Estimate from Historical EvidenceM1 – Estimate from Historical Evidence

Objectively done meta-analysis is importantObjectively done meta-analysis is important What studies to includeWhat studies to include What portions to include (e.g., patient subsets)What portions to include (e.g., patient subsets) Selection to improve strength of meta-analysis and Selection to improve strength of meta-analysis and

applicability to planned new study use of drugapplicability to planned new study use of drug How relevant is historical M1 estimate to a new study:How relevant is historical M1 estimate to a new study:

Relates to issues outside of purely mathematical Relates to issues outside of purely mathematical variance – Not in meta-analysisvariance – Not in meta-analysis

Population unchanged?Population unchanged?Planned eligibility criteriaPlanned eligibility criteriaUnplanned shift in available population related to changes Unplanned shift in available population related to changes

in medical practice (e.g., development of alternative in medical practice (e.g., development of alternative treatments) in intervening timetreatments) in intervening time

Active Control’s treatment regimen unchanged?Active Control’s treatment regimen unchanged? Concomitant care impact changed?Concomitant care impact changed?

Uncertainties Affecting Interpretation of New StudyUncertainties Affecting Interpretation of New Study

Imprecision of control agent’s effect in each of the historical studiesImprecision of control agent’s effect in each of the historical studiesPartially estimated by statistical variance of each studyPartially estimated by statistical variance of each study

Variability of treatment effect size between historical studiesVariability of treatment effect size between historical studiesPartially estimated by meta-analysis methodsPartially estimated by meta-analysis methods

Uncertainty of quantitative extrapolation of historical treatment Uncertainty of quantitative extrapolation of historical treatment effect to treatment effect of control agent in new studyeffect to treatment effect of control agent in new study

The Constancy Assumption – relevance of estimateThe Constancy Assumption – relevance of estimateNot statistically calculableNot statistically calculableCan create an (arbitrary) adjustment inserted into Can create an (arbitrary) adjustment inserted into

mathematical proceduresmathematical procedures Uncertainty of comparison between control and new agent in the Uncertainty of comparison between control and new agent in the

new studynew studyEstimated with statistical variance in new studyEstimated with statistical variance in new studyAS ALWAYS, Statistical approach adequate only if non-AS ALWAYS, Statistical approach adequate only if non-

statistical sources of bias ignorable statistical sources of bias ignorable

Circumstances for Confidence in M1 EstimateCircumstances for Confidence in M1 Estimate Good quality of prior placebo control studies – A&WCGood quality of prior placebo control studies – A&WC Multiple prior placebo controlled trials with comparatorMultiple prior placebo controlled trials with comparator Good grounds for combining data in a meta-analysis (all Good grounds for combining data in a meta-analysis (all

randomized, combinable doses/regimen; outcome randomized, combinable doses/regimen; outcome measure defined, assessed the same way)measure defined, assessed the same way)

Consistent results across all the prior studiesConsistent results across all the prior studies Prior studies done over extended period of time, but Prior studies done over extended period of time, but

completed not long agocompleted not long ago Allows for some changes in populations, concomitant Allows for some changes in populations, concomitant

care, etc, showing this is not criticalcare, etc, showing this is not critical Prior studies done with some variations in designPrior studies done with some variations in design

Implies drug treatment effect not highly sensitive to Implies drug treatment effect not highly sensitive to designdesign

Better with large treatment-associated efficacy effectBetter with large treatment-associated efficacy effect

Historical Knowledge for Non-inferiority PurposesHistorical Knowledge for Non-inferiority Purposes

May be source of difficulty Active control design being used because placebo

control cannotDue to nature of benefit of established drug

May be few (one?) placebo controlled studies with a life-saving drug before it becomes an established, necessary standard of careLimited ability to form precise estimate of treatment

effect

M2 – Clinical ComponentM2 – Clinical Component

How much efficacy is clinically acceptable to give up in the How much efficacy is clinically acceptable to give up in the new agent and still allow new treatment to be medically new agent and still allow new treatment to be medically acceptable to use in place of an already proven active agentacceptable to use in place of an already proven active agent

Clinical judgment, given importance of endpoint, nature of Clinical judgment, given importance of endpoint, nature of disorder, amount of efficacy of the active control agentdisorder, amount of efficacy of the active control agent

Absolute (% of patients, points on scale) or relative to Absolute (% of patients, points on scale) or relative to control’s effect (% of control’s efficacy)control’s effect (% of control’s efficacy)

M2 does not serve to make a hazy M1-statistical analysis M2 does not serve to make a hazy M1-statistical analysis more reassuring by making it more stringent; M2 is from more reassuring by making it more stringent; M2 is from solely clinical meaning point of viewsolely clinical meaning point of view

½ often used½ often used No automatic reason why this is the correct M2 in different No automatic reason why this is the correct M2 in different

settingssettings Each case should be considered independently Each case should be considered independently

Non-inferiority Analysis: Conceptual MethodsNon-inferiority Analysis: Conceptual Methods

Sequential CI methodSequential CI method Dual (Double) CI methodDual (Double) CI method Fixed (Pre-determined) Margin methodFixed (Pre-determined) Margin method

Synthesis methodSynthesis method Putative Placebo methodPutative Placebo method

Strengths and Weaknesses to eachStrengths and Weaknesses to each How confident can ‘we’ be in effect estimates from How confident can ‘we’ be in effect estimates from

historical data?historical data? How many AC studies will we have to consider?How many AC studies will we have to consider? Given nature of benefit, how much risk of being wrong in Given nature of benefit, how much risk of being wrong in

favor of new drug is acceptable?favor of new drug is acceptable? How much risk of being wrong by discrediting new drug is How much risk of being wrong by discrediting new drug is

acceptable?acceptable? Statistical purity vs. discretized logic steps Statistical purity vs. discretized logic steps

Conceptual MethodsConceptual Methods

Sequential CI methodSequential CI method M1 estimate of active comparator’s effect from meta-analysis M1 estimate of active comparator’s effect from meta-analysis

confidence intervalconfidence interval Apply M2 limit to form overall margin (M)Apply M2 limit to form overall margin (M) New agent’s comparative effect estimated with confidence New agent’s comparative effect estimated with confidence

interval from new studyinterval from new study Show C.I. does not exceed overall margin MShow C.I. does not exceed overall margin M

Synthesis methodSynthesis method Combines historical placebo-comparator data and new study Combines historical placebo-comparator data and new study

comparator-test agent data into single calculation.comparator-test agent data into single calculation. Result is numeric value indicating putative placebo-controlled Result is numeric value indicating putative placebo-controlled

efficacy estimate of test agent.efficacy estimate of test agent. Chief difference is conceptual approach of how to address Chief difference is conceptual approach of how to address

inter-study variability and strength of constancy assumptioninter-study variability and strength of constancy assumption

Quality of Study Data & AnalysisQuality of Study Data & Analysis Constancy assumption needs good study qualityConstancy assumption needs good study quality Effects of study flawsEffects of study flaws

Anti-conservative; opposite of superiority studyAnti-conservative; opposite of superiority study FlawsFlaws

Adherence to study protocolAdherence to study protocol Cross-overs Cross-overs Completeness of dataCompleteness of data

Dropouts, missing dataDropouts, missing data Analysis populationAnalysis population

ITT keeps errors in but is statistically pureITT keeps errors in but is statistically pure Per Protocol keeps errors out, but impurePer Protocol keeps errors out, but impure Probably best to do both and show that all ways of Probably best to do both and show that all ways of

looking at dataset are consistentlooking at dataset are consistent

Interpretation of ResultsInterpretation of Results Examples of CIs and assessmentExamples of CIs and assessment Apparent paradox of non-inferiority success with an inferior drugApparent paradox of non-inferiority success with an inferior drug

a b c d e f

Closing CommentsClosing Comments

Size of non-inferiority studiesSize of non-inferiority studies Often large to achieve sufficient precision in estimate of Often large to achieve sufficient precision in estimate of

New-Active effect difference so that CI of comparison New-Active effect difference so that CI of comparison falls within the limited range defined by marginfalls within the limited range defined by margin

When M is small, study is large; when M is large, study When M is small, study is large; when M is large, study can be moderate in sizecan be moderate in size

Biggest factor can be efficacy of active comparatorBiggest factor can be efficacy of active comparator

Equivalence studiesEquivalence studies If this implies greater closeness of effect than “not-If this implies greater closeness of effect than “not-

unacceptably-inferior”, then M is smaller based on M2 unacceptably-inferior”, then M is smaller based on M2 componentcomponent

Consequently sample size of study will increaseConsequently sample size of study will increase

Closing CommentsClosing Comments

Active Control-Noninferiority studies of successive new Active Control-Noninferiority studies of successive new agents ( risk of ‘bio-creep’)agents ( risk of ‘bio-creep’) M based on quantitative data applicable to the M based on quantitative data applicable to the

comparator being used. Risk of using New1, based on comparator being used. Risk of using New1, based on single AC study vs. Std, as the AC agent for New2 single AC study vs. Std, as the AC agent for New2 assessment, and so onassessment, and so on

New1 might be inferior to original comparator by a small New1 might be inferior to original comparator by a small amount (in itself negligible)amount (in itself negligible)

Negligible inferiority added twice (Std-New1; New1-Negligible inferiority added twice (Std-New1; New1-New2) may be not negligible ; frequently only 1-2 study New2) may be not negligible ; frequently only 1-2 study with New1 so that meta-analysis of New1with New1 so that meta-analysis of New1StdStdPbo Pbo gives larger CI, and less assured efficacygives larger CI, and less assured efficacy

If ignored, New3 may have little to no efficacy yet pass If ignored, New3 may have little to no efficacy yet pass the erroneously applied non-inferiority margin the erroneously applied non-inferiority margin

Closing CommentsClosing Comments

Planning and conducting a non-inferiority study does not Planning and conducting a non-inferiority study does not prohibit achieving a conclusion of superiority if such is prohibit achieving a conclusion of superiority if such is the case. Analytic plan of study can allow for this when the case. Analytic plan of study can allow for this when done in the proper manner.done in the proper manner.

Safety issues also always need consideringSafety issues also always need considering Often not as enticing as working on efficacy aspect Often not as enticing as working on efficacy aspect

Reading and Reporting of Active Control StudiesReading and Reporting of Active Control Studies Planning and analysis aspects unique to AC studiesPlanning and analysis aspects unique to AC studies Great difficulty in assessing study result of these Great difficulty in assessing study result of these

aspects not well described in publicationaspects not well described in publication