BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition...

74
BASS XX BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS: COMPARISONS BETWEEN BAYESIAN AND EM ANALYSES 1 Nov, 2013 IBRAHIM TURKOZ, PhD Janssen Research & Development LLC, NJ BASS XX

Transcript of BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition...

Page 1: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS COMPARISONS BETWEEN BAYESIAN AND EM ANALYSES

1 Nov 2013

IBRAHIM TURKOZ PhD

Janssen Research amp Development LLC NJ

BASS XX

BASS XX

Introduction

Background

Literature Review

Objective

Methods

EM Algorithm

Bayesian Approaches

Model Selection

Simulations

Set-up

Results

Application

Discussions and Conclusions

Questions

Outline

Nov 2013 2 Outline

BASS XX

Clinical trials can have a major impact on public health

Required information for designing a clinical trial is not fully

available

Ongoing monitoring of accumulating data is necessary

Interim analyses are common

Usually costly

Requires unblinding treatment assignments

Statistical implications

We will investigate the use of the accumulated data for purposes

of modifying aspects of the study design IN BLINDED

SETTING

Background

3 Introduction Nov 2013

BASS XX

Background Motivating Question MDD Trial[1]

4 Introduction

Additional AD Treatment

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

Nov 2013

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 5 Introduction

Additional AD Treatment

RANDOMIZATION for

those who did not respond

6 weeks long but the primary time

point was Week 4

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 6 Introduction

Additional AD Treatment

RANDOMIZATION for

Patients with Caroll

Depression Score gt20

And CGI-S4

6 weeks long but the primary time

point was Week 4

Outcome Parameters

Clinician Rated Patient Reported

bullHRSD-17 (primary efficacy

scale)

bullCGI-S

bullMost Troubling Symptoms

(MTS) or PaRTS-D

bullPatient Global Impression of

Severity PGIS

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial

Nov 2013 7 Introduction

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 2: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Introduction

Background

Literature Review

Objective

Methods

EM Algorithm

Bayesian Approaches

Model Selection

Simulations

Set-up

Results

Application

Discussions and Conclusions

Questions

Outline

Nov 2013 2 Outline

BASS XX

Clinical trials can have a major impact on public health

Required information for designing a clinical trial is not fully

available

Ongoing monitoring of accumulating data is necessary

Interim analyses are common

Usually costly

Requires unblinding treatment assignments

Statistical implications

We will investigate the use of the accumulated data for purposes

of modifying aspects of the study design IN BLINDED

SETTING

Background

3 Introduction Nov 2013

BASS XX

Background Motivating Question MDD Trial[1]

4 Introduction

Additional AD Treatment

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

Nov 2013

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 5 Introduction

Additional AD Treatment

RANDOMIZATION for

those who did not respond

6 weeks long but the primary time

point was Week 4

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 6 Introduction

Additional AD Treatment

RANDOMIZATION for

Patients with Caroll

Depression Score gt20

And CGI-S4

6 weeks long but the primary time

point was Week 4

Outcome Parameters

Clinician Rated Patient Reported

bullHRSD-17 (primary efficacy

scale)

bullCGI-S

bullMost Troubling Symptoms

(MTS) or PaRTS-D

bullPatient Global Impression of

Severity PGIS

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial

Nov 2013 7 Introduction

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 3: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Clinical trials can have a major impact on public health

Required information for designing a clinical trial is not fully

available

Ongoing monitoring of accumulating data is necessary

Interim analyses are common

Usually costly

Requires unblinding treatment assignments

Statistical implications

We will investigate the use of the accumulated data for purposes

of modifying aspects of the study design IN BLINDED

SETTING

Background

3 Introduction Nov 2013

BASS XX

Background Motivating Question MDD Trial[1]

4 Introduction

Additional AD Treatment

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

Nov 2013

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 5 Introduction

Additional AD Treatment

RANDOMIZATION for

those who did not respond

6 weeks long but the primary time

point was Week 4

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 6 Introduction

Additional AD Treatment

RANDOMIZATION for

Patients with Caroll

Depression Score gt20

And CGI-S4

6 weeks long but the primary time

point was Week 4

Outcome Parameters

Clinician Rated Patient Reported

bullHRSD-17 (primary efficacy

scale)

bullCGI-S

bullMost Troubling Symptoms

(MTS) or PaRTS-D

bullPatient Global Impression of

Severity PGIS

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial

Nov 2013 7 Introduction

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 4: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question MDD Trial[1]

4 Introduction

Additional AD Treatment

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

Nov 2013

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 5 Introduction

Additional AD Treatment

RANDOMIZATION for

those who did not respond

6 weeks long but the primary time

point was Week 4

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 6 Introduction

Additional AD Treatment

RANDOMIZATION for

Patients with Caroll

Depression Score gt20

And CGI-S4

6 weeks long but the primary time

point was Week 4

Outcome Parameters

Clinician Rated Patient Reported

bullHRSD-17 (primary efficacy

scale)

bullCGI-S

bullMost Troubling Symptoms

(MTS) or PaRTS-D

bullPatient Global Impression of

Severity PGIS

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial

Nov 2013 7 Introduction

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 5: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 5 Introduction

Additional AD Treatment

RANDOMIZATION for

those who did not respond

6 weeks long but the primary time

point was Week 4

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 6 Introduction

Additional AD Treatment

RANDOMIZATION for

Patients with Caroll

Depression Score gt20

And CGI-S4

6 weeks long but the primary time

point was Week 4

Outcome Parameters

Clinician Rated Patient Reported

bullHRSD-17 (primary efficacy

scale)

bullCGI-S

bullMost Troubling Symptoms

(MTS) or PaRTS-D

bullPatient Global Impression of

Severity PGIS

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial

Nov 2013 7 Introduction

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 6: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question MDD Trial[1]

Nov 2013 6 Introduction

Additional AD Treatment

RANDOMIZATION for

Patients with Caroll

Depression Score gt20

And CGI-S4

6 weeks long but the primary time

point was Week 4

Outcome Parameters

Clinician Rated Patient Reported

bullHRSD-17 (primary efficacy

scale)

bullCGI-S

bullMost Troubling Symptoms

(MTS) or PaRTS-D

bullPatient Global Impression of

Severity PGIS

Placebo + AD Treatment as usual Placebo + AD Treatment as usual

Active + AD Treatment as usual

1 Mahmoud AR Pandina JG Turkoz I Kosik-Gonzalez C Canuso MC Kujawa JM and Gharabawi GM Risperidone for treatment-refractory major depressive disorder

Annals of Internal Medicine 2007 147 593-602

Prior AD Treatment

BASS XX

Background Motivating Question MDD Trial

Nov 2013 7 Introduction

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 7: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question MDD Trial

Nov 2013 7 Introduction

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 8: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 8 Introduction

Psychosis

Mania Depression

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 9: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 9 Introduction

Psychosis

Mania Depression

PANSS

30-210

YMRS

0-60

HAM-D-21

0-52

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 10: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question Schizoaffective Trial[1]

Nov 2013 10 Introduction

BL Wk 6

n=95 PBO

n=216 PALI PR 3 - 12 mgd (start at 6 mg flexible)

Study 3002

N=311

Psychosis

Mania Depression

1 Canuso CM Lindenmayer JP Kosik-Gonzalez C Turkoz I Carothers J Bossie CA Schooler NR A randomized double-blind placebo-controlled study of 2 dose ranges of

paliperidone extended-release in the treatment of subjects with schizoaffective disorder J Clin Psychiatry 2010 May71(5)587-98

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 11: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 11 Introduction

Psychosis

Mania

Depression

Changes in PANSS

Total Score at Week 6

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 12: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 12 Introduction

Psychosis

Mania

Depression

YMRS

HAM-D-21

PRIMARY ENDPOINT SECONDARY ENDPOINTS

FOR REGISTRATION PURPOSES

Changes in PANSS

Total Score at Week 6

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 13: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question Schizoaffective Trial

Nov 2013 13 Introduction

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 14: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 14 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 15: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background Motivating Question ADHD Trial 1

Nov 2013 15 Introduction

1 Wigal SB Wigal T Schuck S Brams M Williamson D Armstrong RB and Starr HL Academic Behavioral and Cognitive Effects of OROSreg Methylphenidate on Older Children

with Attention-DeficitHyperactivity Disorder J Child Adolesc Psychopharmacol 2011 April 21(2) 121ndash131

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 16: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background

Nov 2013 16 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 17: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background

Nov 2013 17 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 18: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background

Nov 2013 18 Introduction

It is important to evaluate secondary endpoints to differentiate your

drug from other drugs in market place

Another symptom

Functionality

Medication Satisfaction

Certain Adverse Reactions

Cognition

Etc

Failure to consider important outcomes can limit the

conclusions of clinical trials

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 19: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Background

Nov 2013 19 Introduction

bull Examination of appropriate statistical methods to test secondary

endpoints requires careful consideration

bull Fixed sequence gate-keeping procedures [1234] are widely used

because of their ease of application and interpretation in many

circumstances

bull These gate-keeping procedures often require prospective ordering

of null hypotheses for secondary endpoints

bull Can we use information that is available to order the null

hypotheses based on interim effect sizes without breaking the

blind

1 Demitrienko A Tamhane A Bretz F Multiple testing problems in pharmaceutical statistics 2010 ChapmanampHallCRC Biostatistics Series

2 Westfall PH Krishen A Optimally weighted fixed sequence and gatekeeper multiple testing procedure J of Statistical Planining and Inference 99 (2001) 25-40

3 Weins B Dmitrienko A (2005) The fallback procedure for evaluating a single family of hypotheses Journal of Biopharmaceutical Statistics 15 929ndash942

4 Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Stat in Med 2009 28586ndash604

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 20: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Literature Review

Nov 2013 20 Introduction

bull EM algorithm to estimate the within group variance for sample size re-

estimation without unblinding at interim stages has been suggested [123]

bull Enrollment order of subjects and the randomization block sizes to estimate the

within group variance have also been used [4]

bull Risk of unblinding the treatment effect from blinded inferences based on

knowledge of the randomization block size for continuous and binary

outcomes [5]

bull The recent draft FDA guidance on adaptive designs [6] discusses possible

study design modifications such as selection andor order of secondary

endpoints in addition to sample size re-estimation

bull Blinded treatment effects for survival endpoints[7] were also examined

1Gould AL Shih WJ Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance Communications in Statistics Theory and Methods

1992 21 2833ndash2853

2 Shih WJ Sample size reestimation in clinical trials In Biopharmaceutical sequential statistical applications Peace K (ed) Marcel Dekker New York 1992 285ndash301

3 Shih WJ Sample size reestimation for triple blind clinical trials Drug Information Journal 1993 27 761ndash764

4 Xing B Ganju J A method to estimate the variance of an endpoint from an on-going blinded trial Statistics in Medicine 2005 24 1807-1814

5 Miller F Friede T Kieser T Blinded assessment of treatment effects utilizing information about randomization block length Statistics in Medicine 2009 28 1690-1706

6 Guidance for industry Adaptive design clinical trials for drugs and biologics

httpwwwfdagovdownloadsDrugsGuidanceComplianceRegulatoryInformationGuidancesucm201790pdf

7 Xie J Quan H Zhang J Blinded assessment of treatment effects for survival endpoint in an ongoing trial Pharmaceutical Statistics 2012

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 21: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Objective

Nov 2013 21 Introduction

bull Instead of formal interim analyses on unblinded data use the

information from blinded data

bull Assess the feasibility of estimating magnitude of treatment effect

on various secondary endpoints in ongoing trial

bull Primary objective is to compare posterior ordering of the signal-

to-noise ratios (effect sizes) using available data

points

d

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 22: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Model

Nov 2013 Methods 22

( 1 01 12 ) is the response variable for the endpoint from

20 ~ (0 )

th thjijkY i n j k K k j

ijk k k ik ik ijk ky z N i i d

20 0

21 1

0

treatment group on subject

~ ( )

~ ( )

the mean placebo response

magnitude of the treatment effect

latent binary treatment assignment

We assume that lower scores a

i k k k

i k k k

k

k

ik

i

Y N

Y N

z

y

21 0~ ( )

re better and the average effect of the treatment is

k k k k

k

Y N

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 23: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Model

Nov 2013 Methods 23

For a two component univariate normal mixture model it is appropriate to specify the

complete likelihood function for a given response in the form

( |p y z

2 20 0 1 1 0

1 0 =p(z 1)

where ( ) ( ) is randomization weight

The signal-to-noise ratio for endpoint ( 1 )

) ( | ) (1 ) ( | )

Our objective is

k k kk d k K

p y p y

1 0

θ θ

θ θ θ

1 2

to compare the EM and Bayesian algorithm induced

posterior ordering of the signal-to-noise ratios of endpoints ( | )kP d d d y

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 24: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

EM Algorithm

Nov 2013 Methods 24

=p(z 1)

MMLEs are the most common way to estimate parameters of interest however MMLEs

often do not have closed form solutions EM seeks the MMLE by iterativel

1 0( | ) ( | ) (1 ) ( | )p y z p y p y 1 0θ θ θ

( )

y applying

Expectation and Maximizing steps

ˆE-Step log ( | ) |

M-Step the conditional expectation is maximized with respect to This yields the new estimates

tE p y z yθ θ

θ

( 1) ( 1)ˆ for and a distribution for t tz θ

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 25: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

EM Algorithm

Nov 2013 Methods 25

( ) ( ) 2( )( 1) ( ) 0

( ) ( )( ) 2( ) 2( )0 0

Conditional expectation of latent treatment assignment at iteration ( 1) is

ˆˆ ˆ( | )ˆ( 1| )

ˆˆ ˆ ˆ ˆ( | ) (1 ) ( | )

In the M step the mean pa

t t tt t

i t tt t t

t

N yP z y

N y N y

θ

20

( 1) ( 1) ( 1)( ) ( )0

( 1) ( 1)1 10

( 1) ( 1)( ) ( )

1

ˆˆ ˆrameters and variance parameter at iteration are ( 1)

ˆ ˆ ˆ( 0 | ) ( 1| )( )

ˆˆ

ˆ ˆ( 0 | ) ( 1| )

n nt t tt t

i ii it ti i

nt tt t

i i

i i

t

P z y y P z y y

P z y P z y

θ θ

θ θ

1)

1

( 1) ( 1) ( 1) ( 1)( ) 2 ( ) 21 0

2(t 1

ˆ ˆˆ ˆ( 1| )( ) ( 0 | )( )

ˆ

n

nt t t tt t

i ii i

i

P z y y P z y y

n

θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 26: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 26

The likelihood function of the mixture distribution and the prior distributions of and z

are combined to obtain the joint posterior distribution which is assumed to contain all the information

abou

θ

t the unknown parameters Using the Bayes theorem the posterior distribution satisfies

where

( | ) is the likelihood

( | ) is the prior probability of latent tre

( | ) ( | ) ( | ) ( )

p y z

p z

p z y p y z p z p

θ

θ

θ θ θ θ

atment assignment

( ) is the prior distribution of p θ θ

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 27: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 27

The conditional distribution of the treatment allocation is proportional

this reduces to latent treatment allocation probability in EM

Prior Distribution

( | ) ( | ) ( | ) ( )

( | ) ( | )

p z y p y z p z p

p z y p y z

θ θ θ θ

θ θ

20 0 0

21 0 1 1

2

s

in order to reduce sensitvity

For simplicity purposes we have

~ ( )

~ ( )

~ ( )

( )

j N

N y s

N y s

Gamma

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 28: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 28

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 29: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 29

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for treatment

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 30: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 30

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Likelihood for placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 31: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 31

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for treatment arm

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 32: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 32

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

i i

i i

z z

y

y y

s s

Prior for Placebo arm

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 33: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 33

Regardless of the structure of the prior distribution the posterior distribution does not have

a simple closed form In order to make inferences about the unknown parameters MCMC

methodologies are em

2 2 20 0 0

2

2 2

ployed to generate samples from the posterior distribution The joint

posterior probability distribution of these parameters satisfies

( | ) ( | ) ( ) ( )

1 1exp (

2

n

p z y p y z p z p

y

2 20 02

1 0

12 220 1 0 0

2 2 21 0

1) exp ( )

2

( ) ( ) 1exp exp exp( )

2 2

In computations =3 and small scale hyperparameter ( 001) are assumed

i i

i i

z z

y

y y

s s

Prior for Sigma

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 34: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach

Nov 2013 Methods 34

0 12 2 2

0 110

2 2 2 2 2 20 1 0 1

0

1 0 12

The posterior conditional distributions of parameters of interest in equation are given by

1~

1 1 1 1

( )

~

i i

i

i i

i

y z y y

s sN

n n

s s s s

y zy

sN

21

1 12 2 2 2

1 1

20

2 12

3

1

1 1

( )

~

i i

i i

i i

i

n

z z

s s

y z

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 35: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 35

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 36: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 36

We assume a trial with stratified randomization The sample is divided into subsamples on the

basis of stratifaction factor (covariate additional medication usage and groups)

Let denote ti

yes no

g

22

0 2

he known strata membership of the subject 1 if observation belongs to the

strata and 0 if it does not The joint posterior distribution then takes the form

1( | ) e

i

n

yes no

i g yes

p z y

202

1 01

2 20 12 0 0

02 2 21 00 01

12

2

1xp ( )

2

( ) ( )1exp ( ) exp exp

2 2 2

1exp( )

i

i i

i i

i i

i g

z g

g gi

z g

y

y yy

s s

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 37: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 37

202

2 20 02 2

02

Conditional posterior probability treatment assignment at each iteration is given by

1exp ( )

2 Yes strata

1 1exp ( ) exp ( )

2 2

( 1)

1exp (

2

i yes

i yes i

i

i

y

y y

P z

y

2

2 20 02 2

)

No strata1 1

exp ( ) exp ( )2 2

no

i no iy y

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 38: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate

Nov 2013 Methods 38

1 102 2 2

0 110

2 2 2 2 2 20 1 0 1

0

The posterior distribution for parameters of interest are computed to be

( ) ( ) ( )

1~

1 2 1 2

( )

~

i i yes no yes yes no no

i

i

yes

y z y yy

s sN

n n

s s s s

y

N

0

1 1 0 1 1 2 0 1

2 2 2 21 1

1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 21 1 1 1

20 0

2

( )

1 1 ~

1 1 1 1

( ) (

~

i i i

i g yes i g no

noi i i i

i g i g i g i g

i i yes i i

z y zy y

s sN

z z z z

s s s s

y z y z

2

1 1 1 2

23

)

no

i g i g

n

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 39: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 39

0

[123]0

0 0

0

Denote the historical data by and current data by

[01] to control the influence of the historical data on the current study

1 approaches full barrowing from

0 approaches no ba

D D

D

0

0

0

rrowing from

The basic idea is to use the power parameter to control the influence of the historical data on the current study

Let assume

( | ) is the past data likelihood

D

L D

0 0 0 0 0

0 0

( | ) is the current data likelihood

( | ) ( | ) ( ) ( ) is the posterior given past likelihood

Then the full likelihood is proportional to ( | ) ( | )

L D

p D L D p p

p D L D

1 Ibrahim J Chen MH Power prior distributions for regression models Statistical Science 2000 1546-60

2 Duan Y Ye K Smith EP Evaluating water quality using power priors to incorporate historical information Environmetrics 2006 17 95-106

3 Hobbs BP Carlin BP Mandrekar S and Sargent DJ Hierarchical commensurate and power prior models for adaptive incorporation of historical information in

clinical trials Biometrics 2011 671047-1056

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 40: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 40

0

0 0

22

0 0 2 2

The joint posterior distribution can be extended to include the prior for

Think of as the historical effective sample size

The result takes the form

1 1( | ) exp (

2

n

y n

n

p z y y

0 0

0 0

0

20

1

21 0 1 1 2

02 21 10

21 0 1 1

21 1

0 0 0

0

)

( )1exp ( ) exp

2 2

( )exp

2

( )exp

i

i

i g

z g

p yes yes yes p yesi

z

p no no no p no

p

n y ny

s s

n y n

s s

n y

s

0

2 10 2

2 20

1 10 0

1exp( )

2

(1 )

p

a b

n

s

Prior for Power Parameter

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 41: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 41

0

0

1 2 20 0 0 1 1 12

0 00 2 20 1

0 1

The posterior distribution of parameters of interest is computed to be

( ) ( )(1 ) exp

2

( ) exp

2

aop yes yes p yesb

no no

y n y n

s s

y

0 0 0

0 0 0 0

21

21

0 0 1 1 1 1

2 2 20 11

00 1 0 1

2 2 2 2 2 20 1 0 1

( ) ( )( )

1~

p no

p yes yes p yes no no p noi i yes no

i

p p p p

n

s

y n y n y ny z

s sN

n n n nn n

s s s s

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 42: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Bayesian Approach with Covariate and a Power Prior

Nov 2013 Methods 42

0

0 0

0

0 1 1 1 1

2 21

1 1 1 1 1 1

2 2 2 21 1

The posterior distribution of parameters of interest is computed to be

( )( )

1~

(

~

i i

yes p yesi g

yesi i

p yes p yesi g i g

no

y zy n

sN

z zn n

s s

y

N

0

0 0

0

0 1 1 1 2

2 21

1 1 1 2 1 2

2 2 2 21 1

2 20 0

1 1 1 22

23

)( )

1

( ) ( )

~

i i

no p noi g

i i

p no p noi g i g

i i yes i i no

i g i g

n

zy n

s

z zn n

s s

y z y z

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 43: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Model Selection

Nov 2013 Methods 43

By contrast with frequentist analogues Bayesian model comparison distinguishes which likelihood and prior

combinations better fit the data These measures of performance can be used to choose a singl

[1]

e ldquobestrdquo model

or improve estimation via model averaging

The deviance information criterion (DIC) provides a natural measure of performance in this setting

Models with smaller DIC should be preferre

0 0

0 0

d to models with larger DIC

=2 ( ) since ( )

ˆˆ =2 ( ) ( )

ˆˆ 2 ( ) ( )

d

d

DIC p D

D D p D D

D D

Expectation over Expectation over

θ θ

1 DJ Spiegelhalter NG Best BP Carlin A Linde Bayesian measures of model complexity and fit (with discussion) Journal of the Royal Statistics

Society Series B (Statistical Methodology)2002 64 583ndash639

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 44: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Model Selection

Nov 2013 Methods 44

0

20

0 021

2

0

( )( ) this is a posterior expectation over ( )average after all the iterations

ˆ

ˆ averaged after all the iterations

updates at each iter

0 0ˆˆ2 ( ) ( )

i i

i

y zD E

DIC D D

20

0 21

ation

updates at each iteration

treatment assignment at each iteration

ˆˆ( )ˆˆ( ) this is a posterior expectation over it is using averaged parametersˆ

ˆ

i

i

i iz i

i

z latent

y zD E z

2

0

averaged after all the iterations

ˆ averaged after all the iterations

ˆ averaged after all the iterations

treatment assignment at each iterationiz latent

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 45: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

MCMC Gibbs Sampling

Nov 2013 Methods 45

An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted

The main drawback of the Gibbs sampler in this setting is the lack of mixing

We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling

This involved subsampling 10 of the treatment and placebo group observations and proposing a label switch

1) Draw random candidate samples fromfor switching treatment

2) Generate from the Uniform (01) distribution

(1 ( ( 1))3) If make label change otherwise return to s

( 1)

i i

i i

u

p zu

p z

tep 1

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 46: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Set-Up

Nov 2013 Simulations 46

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt ~0 170

4 10 gt 05 gt ~0 72

5 10 gt 05 gt ~0 468

1d 3d2d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 47: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Set-Up

Nov 2013 Simulations 47

Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=005

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 48: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Set-Up

Nov 2013 Simulations 48

1 2 3

2 treatment arms with a 11 treatment allocation ratio

3 uncorrelated endpoints and

No missing data

For each case 20 data sets are generated ( 1 20)

Different prior for each of

k y y y

l l

the endpoints

20 0 0

21 0 1 1

2

2 20 0 1

Generate Input Data Sets

( ) ~ ( )

( ) ( ) ( ) ~ ( )

~ ( ) 123 1 20

Without loss of generality we assumed that

10 10

k k

k k

k k

k k k

k l N Y s

k l k l k l N Y s

IG b k l

Y s s

Scenario N

1 05 = 05 = 05 170

2 08 gt 05 gt 03 170

3 10 gt 05 gt 0 170

4 10 gt 05 gt 0 72

5 10 gt 05 gt 0 468

2d 3d1d

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 49: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Set-Up

Nov 2013 Simulations 49

2 20 1 Each true input clinical trial data set is based on ~ (1010 ) and with normal distribution with variance 10

and corresponding mean value to generate effect sizes for a given scenario

Initi

N

20

0

al values of ( ) for simulations are based on the true input data set parameters and the initial

power parameter estimate is given as 025 and 05

Posterior inferences are based on 1000 i

th

terations

The first 200 iterations are considered to be the burn in period for both EM and Bayesian algorithms

Every 4 point after the burn in period is stored for the Bayesian simulations to re

duce correlations

to emulate thinning process

Eight hundred simulation results in EM and 200 results in Bayesian methods are summarized

Posterior parameter estimates are summuarized

Empirical pr

obability of each triplet combination of effect size ordering is examined

In each case for a given data set the last 20 simulation results are kept for effect size orderings

For each of the study data sets a total of 20 20 20 8000 possible triplet combinations of orderings are available

For 20 different study data sets a total of 160000 possible orderings are available

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 50: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results

Nov 2013 Simulations 50

Posterior Probability of Effect Size Ordering by Scenario

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 51: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results

Nov 2013 Simulations 51

Posterior Distribution of Effect Sizes by Scenario Averaged Over 20 Data Sets

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 52: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results

Nov 2013 Simulations 52

Posterior Distribution of Effect Sizes by Scenario

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 53: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 53

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 54: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 54

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 55: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 55

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 56: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 56

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 57: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 1

Nov 2013 Simulations 57

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 58: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 58

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 59: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 2

Nov 2013 Simulations 59

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 60: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 60

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 61: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 3

Nov 2013 Simulations 61

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 62: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 62

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 63: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 4

Nov 2013 Simulations 63

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 64: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 64

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 65: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Simulation Results Scenario 5

Nov 2013 Simulations 65

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 66: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

MDD Trial

Nov 2013 Application 66

Primary Time Point

Week 4 or Week 6

Primary Endpoint

HRSD-17 or PaRTS-D

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 67: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

MDD Trial Results

Nov 2013 Application 67

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 68: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

MDD Trial Results

Nov 2013 Application 68

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 69: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

MDD Study Simulations

Nov 2013 Application 69

4 6

Decision Rule 1

Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6

( ( ) ( ) | )

Decision Rule 2

Calculate 95 Credible Intervals for W

Wk WkP d H d H

Y

4 4 4

6 6 6

4

2 204 4 0 0

14 4

eek 4 and Week 6 effect sizes

( ) ( ) ( )

( ) ( ) ( )

Priors for HRSD 17 placebo and treatment means at Week 4 ( ) were assumed to be

( ) ~ ( 15 7 )

( ) ~

Wk Wk Wk

Wk Wk Wk

Wk

Wk

Wk

d H d H d H

d H d H d H

H

H N Y s

H

2 21 1( 12 7 )N Y s

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 70: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

MDD Study Simulations

Nov 2013 Application 70

Summary of Effect Sizes for HRSD-17 and PaRTS-D

At Week 4 EM Results were Not Available

Posterior Ordering of Effect Sizes

for HRSD-17

Week 4 Week 6

EM 00 1000

Bayesian 477 523

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 71: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

MDD Study Simulations

Nov 2013 Application 71

DIC Scores on HRSD-17 and PaRTS-D

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 72: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 72

We investigated whether or not the researchers can obtain reliable estimates of effect sizes

without knowing treatment assignments

Instead of relying on the clinicianrsquos subjective evaluations the

suggested methodologies

provide numerical assistance

We showed how to order secondary null hypotheses while the study is ongoing

without unblinding the treatments

without losing the validity

of the testing procedure

and with maintaining the integrity of the trial

In our simulations we used prior distributions whose hyper parameters reflect the true values

of the model parameters I

n MDD analyses we used prior distributions whose hyper parameters

reflect the original study design elements

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 73: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Discussion and Conclusion

Nov 2013 Discussion 73

The EM algorithm frequently failed to generate posterior parameter estimates

Both the Bayesian and the EM algorithms overestimated treatment differences and

standardized effect sizes

The Bayesian

algorithms performed better than existing EM algorithm counterparts in ordering

the standardized effect sizes

With the Bayesian algorithm the posterior probability for identifying the ground-truth ordering

increased both as a function of the effect size differences and as a function of the sample size

For large sample sizes the proportion of times the true ordering was selected was high (ab ove 35)

and the variability of standardized effect sizes was low

With the EM algorithm the probability for identifying the ground-truth ordering was low

(sometimes near zero) and the variability of

standardized effect sizes was high

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS

Page 74: BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS ... 2013 Turkoz.pdf · endpoints in addition to sample size re-estimation • Blinded treatment effects for survival endpoints[7]

BASS XX

Discussion

Nov 2013 Discussion 74

THANK YOU

QUESTIONS