Statistical considerations in small proof-of-concept trials, including crossover designs
description
Transcript of Statistical considerations in small proof-of-concept trials, including crossover designs
(c) Stephen Senn 2008 1
Statistical considerations in small proof-of-concept trials, including
crossover designs
Stephen Senn
(c) Stephen Senn 2008 2
• People look down on marketing men
• It’s not true that they are not scientists
• They work in sell biology
• I would like to take this opportunity to draw your attention to a book I rather like
• In fact I wrote it myself
(c) Stephen Senn 2008 3
Outline
• Decision analysis and proof of concept
• Value of information perspective
• Place of cross-over trials
• Carry-over
• The potential for cross-over trials in studying individual response
(c) Stephen Senn 2008 4
A Model
1
C
E
E
C
C E
C
C
R
f
f
V f f
Probability proof of concept (POC) study successful
Probability proof of efficacy study (POE) successful if POC successful
Probability POE study successful if POC unsuccessful
Probability POE study successful
Cost of POC including any lost sales through extra delay
Cost of POE study
Expected NPV revenue if POE initiated immediately and successful
Value of strategy of POE only
Value of strategy of POC + POE
Value of POC study
(c) Stephen Senn 2008 5
Model Continued
max 0, 1
max 0, max 0, 1 max 0,
E E
C E E C
f R C
f R C R C C
(c) Stephen Senn 2008 6
Example100, 5, 25, 0.3, 0.25,
1C ER C C
Value of two strategies plotted against , the probability POE successful if POC successful
0 0.5 1
5
direct POEinitial POC
Expected return on two strategies
Prob POE successful if POC success
0.65
(c) Stephen Senn 2008 8
Value of Biomarker Information in Terms of Posterior Variance
• Suppose that over all products for this indication the correlation of true therapeutic and biomarker outcomes is 0.9
• Let the prior means be zero in this class• Let the prior variances be 1• Let the data variance of a minimal
experiment be also 1– Implies prior information equivalent to one
minimal experiment
(c) Stephen Senn 2008 9
Here n is the number of minimal experiments we run
Of course we expect a biomarker experiment to be cheaper than a therapeutic one
Nevertheless note that fairly rapidly there is no interest in getting further biomarker information
Posterior variances based on proof of concept trial
10 30 50
0.6
0.5
0
0.4
0.3
0.2
20
0.1
0.0
40
n
Poste
rior va
rian
ce
simulated therapeuticsimulated biomarkertheory therapeutictheory biomarker
(c) Stephen Senn 2008 10
A Serious Warning to Statisticians
In the mathematical formulation of any problem it is necessary to base oneself on some appropriate idealizations and simplification. This is, however, a disadvantage; it is a distorting factor which one should always try to keep in check, and to approach circumspectly. It is unfortunate that the reverse often happens. One loses sight of the original nature of the problem, falls in love with the idealization, and then blames reality for not conforming to it.
De Finetti 1975
‘The only way that human beings could ever have survived as a species for long as we have is that we’ve developed another kind of decision-making apparatus that’s capable of making very quick judgements based on very little information.
Malcolm Gladwell, Blink, 2005
(c) Stephen Senn 2008 11
My Gloomy Take on This
• We don’t really understand this topic• There may be less value in proof of
concept studies than we propose• Therapeutic studies may be valuable even
if they have low power• There is no point in undertaking POC
studies unless you can see circumstance under which they would cause you to cancel projects
(c) Stephen Senn 2008 12
Appropriate Attitudes for Cross-over Trials
• They are not suitable for all indications and questions
• They are extremely valuable for some indications and questions
• Carry-over has to be dealt with by washout• Don’t pre-test for carry-over• Don’t rely on classical statistical approaches to
carry-over• Cross-over trials have great potential in
investigating individual response
(c) Stephen Senn 2008 13
Carry-over
Definition: Carry-over is the persistence (whether physically or in terms of effect) of a treatment applied in one period in a subsequent period of treatment.
If carry-over applies in a cross-over trial we shall, at some stage, observe the simultaneous effects of two or more treatments on given patients.
We may, however, not be aware that this is what we are observing and this ignorance may lead us to make errors in interpretation.
(c) Stephen Senn 2008 14
The simple carry-over model.
This is a very popular model amongst “applied” statisticians of a mathematical bent.
The model assumes that if a carry-over effect is present
1) it lasts for one period exactly
2) it depends on the engendering treatment only and not on the perturbed treatment.
(c) Stephen Senn 2008 15
Three Period Bioequivalence Designs
• Three formulation designs in six sequences common.
• Subjects randomised in equal numbers to six possible sequences. – For example, 18 subjects, three on each of the
sequences ABC, ACB, BAC, BCA, CAB, CBA. – A = test formulation under fasting conditions, – B = test formulation under fed conditions – C = reference formulation under fed conditions.
(c) Stephen Senn 2008 16
Period
Sequence 1 2 3
ABC A 0 B 1/6 C -1/6
ACB A 0 C -1/6 B 1/6
BAC B 1/6 A 0 C -1/6
BCA B 1/6 C -1/6 A 0
CBA C -1/6 A 0 B 1/6
CAB C -1/6 B 1/6 A 0
Weights for the Three Period Design:
not Adjusting for Carry-over
(c) Stephen Senn 2008 17
Properties of these weights
• Sum to 0 in any column, – eliminates the period effect.
• Sum to 0 in any row – eliminates patient effect
• Sum to 0 over cells labelled A– A has no part in definition of contrast
• Sum to 1 over the cells labelled B and to -1 over the cells labelled C– Estimate contrast B-C
(c) Stephen Senn 2008 18
Period
Sequence 1 2 3
ABC A -1/24 Ba 4/24 Cb -3/24
ACB A 1/24 Ca -4/24 Bc 3/24
BAC B 4/24 Ab 2/24 Ca -6/24
BCA B 5/24 Cb -2/24 Ac -3/24
CBA C -4/24 Ac -2/24 Ba 6/24
CAB C -5/24 Bc 2/24 Ab 3/24
Weights for the Three Period Design:
Adjusting for Carry-over
B-C contrast: illustration of treatment effect and elimination of period and patient effects
(c) Stephen Senn 2008 19
Period
Sequence 1 2 3
ABC A -1/24 Ba 4/24 Cb -3/24
ACB A 1/24 Ca -4/24 Bc 3/24
BAC B 4/24 Ab 2/24 Ca -6/24
BCA B 5/24 Cb -2/24 Ac -3/24
CBA C -4/24 Ac -2/24 Ba 6/24
CAB C -5/24 Bc 2/24 Ab 3/24
Weights for the Three Period Design:
Adjusting for Carry-over
Illustration of elimination of ‘carry-over’ effects
(c) Stephen Senn 2008 20
Have We Got Something for Nothing?
• Sum of squares weights of first scheme is 1/3 (or 4/12)
• Sum of squares of weights of second scheme is 5/12
• Given independent homoscedastic within- patient errors, there is thus a 25% increase in variance
• Penalty for adjusting is loss of efficiency
(c) Stephen Senn 2008 21
The difference between mathematical and applied statistics is that the former is full of lemmas whereas the latter is full of dilemmas
(c) Stephen Senn 2008 22
The Dangers of Pre-testing
• Situation with AB/BA design– Two-stage procedure is very badly biased– CARRY and PAR are highly correlated
• 1/2 < < 1
• Three treatment design– Same problem– Carry-over and adjusted estimates correlated
= 0.45
(c) Stephen Senn 2008 23
The Phoenix Bioequivalence Trials
• Analysed by D’Angelo, Potvin & Turgeon *
• 20 drug classes
• 1989-1999
• 12 or more subjects
• 96 three period designs
• 324 two period designs
D'Angelo, G.Potvin, D.Turgeon, J. J Biopharm Stats, 11, 27-36, 2001
(c) Stephen Senn 2008 24
AUC Cmax
0 : 115567899 1 : 01458999 2 : 01225568999 3 : 011335577 4 : 24688 5 : 35667788 6 : 00336667888 7 : 14444566999 8 : 011233468888 9 : 13335667899
0 : 223557888 1 : 4677799 2 : 000124566899 3 : 011124689 4 : 01223455799 5 : 00045599 6 : 000166667778 7 : 0345566779 8 : 2345779 9 : 13444556889
Three Treatment Designs
P-Values for Carry-Over
“Significant” results in bold
Senn, S. J., G. D'Angelo, et al. (2004). "Carry-over in cross-over trials in bioequivalence: theoretical concerns and empirical evidence." Pharmaceutical Statistics 3(2): 133-142.
(c) Stephen Senn 2008 25
AUC Cmax
0 : 00111111222222234444 0 : 5666777777789999 1 : 00000112222223333 1 : 5556667777899999 2 : 0011112223344444 2 : 555666788899999 3 : 00001112233344 3 : 5556666666777778888899999 4 : 001111112222223334 4 : 55666666777777788999 5 : 00000111222333344444 5 : 566677888899 6 : 000001134 6 : 55666667777888889999 7 : 111233333344 7 : 555556777888899 8 : 0000112234444 8 : 55666778888999 9 : 00011112233334444 9 : 555567777788999
0 : 00122222344 0 : 55555556666677999999 1 : 0001122233333344444444 1 : 55566667778888899 2 : 00011111122344 2 : 566667788889999 3 : 111112222233444444 3 : 555566666777778888999 4 : 000001112222333334444 4 : 5557888889999 5 : 00001122233 5 : 5555666678999 6 : 0000111222233334 6 : 55555566677788889999 7 : 000000112223344 7 : 6666777777889 8 : 0122233444 8 : 55666677888899 9 : 1111111222333444 9 : 555555556666677778889999
Two Treatment Designs
“Significant” results in bold
(c) Stephen Senn 2008 26
StudyDesign
Variable Totalnumber
of studies
KSstatistic
p-value*
2-way AUC0-t 324 0.0645 0.1354Cmax 324 0.0496 0.4040
3-way AUC0-t 96 0.1048 0.2424Cmax 96 0.0542 0.9407
*H0: true cdf U[0,1] vs. H1: true cdf NOT U[0,1]
Test of Uniformity of P-Values
(c) Stephen Senn 2008 27
Galling as this may appear to statisticians, the cure for carry-over is more biological and pharmacological understanding not more statistics
(c) Stephen Senn 2008 28
Conclusions
• Distribution of P-values uniform– no evidence of carry-over
• Carry-over a priori implausible– presence testable by assay
• No point is testing for it– leads to bias
• Or adjusting for it– increased variance
(c) Stephen Senn 2008 29
Possible Strategy
• Run multi-period cross-overs
• Patient by treatment interaction becomes identifiable
• This provides an upper bound for gene by treatment interaction– Because patients differ by more than their
genes
(c) Stephen Senn 2008 30
Second cross-over
Responders Non-Responders
Total
First cross-over
Responders 24 0 24
Non-Responders
0 8 8
Total 24 8 32
(c) Stephen Senn 2008 31
Second cross-over
Responders Non-Responders
Total
First cross-over
Responders 18 6 24
Non-Responders
6 2 8
Total 24 8 32
(c) Stephen Senn 2008 32
Advantages and DisadvantagesPRO CON
• Cheap• Low tech• Insight into sources
of variation gained
• Only suitable for chronic diseases
• Demanding of patient’s time
• Unglamorous• Does not produce
diagnostic patents
(c) Stephen Senn 2008 33
An Overlooked Source of Genetic Variability
• Humans may be classified into two important genetic subtypes.
• One of these suffers from a massive chromosomal deficiency.
• This is expressed in.– Important phenotypic differences.– A massive disadvantage in life expectancy.
• Many treatment strategies take no account of this.
• The names of these subtypes are...