Post on 13-Jan-2016
description
Revisiting an Old Topic:Probability of Replication
D. Lizotte, E. Laber & S. Murphy
Johns Hopkins Biostatistics
September 23, 2009
2
Outline
• Scientific Background
• Our Estimand: Probability of Selection
• Estimators
• STAR*D
• Where to go from here?
3
Scientific Background
First experiment results in– or – ,
– what is the chance that we will replicate this result in a subsequent experiment?
– Prob. of Concurrence or Prob. of Replication– Killeen (2005) followed by great controversy in
psychology (Cumming, (2005, 2006, 2008); MacDonald (2005);Doros & Geier(2005); Iverson(2008); Iverson, Wagenmakers & Lee (2008); Asby & O’Brien(2008), Iverson, Lee & Wagenmakers (2009)……)
4
Scientific Background
Similar problem but discredited:
• Post-hoc power/ Observed power: Assuming the observed standardized effect size is the truth, calculate the probability of rejecting null hypothesis. Hoenig & Heisey (2001)
5
Scientific Background
First experiment results in– or – – what is the chance that we will replicate this result
in a subsequent experiment?
• Why is this question so attractive?• Scientists (including statisticians!) often want to
answer this question with 1 – p-value
6
Scientific Background
• First experiment results in– or – ,– what is the chance that we will replicate this result
in a subsequent experiment?
• 1 – p-value does not address this question.– Goodman (1992), Cumming (2008)– 1 – p-value is not an estimator.
7
Scientific Background
• Much confusion about estimand:– , what is the chance that we will replicate this
result in a subsequent experiment?• Do we want to “estimate”
1) or
2) or
3) or
4) ?
• Good frequentist properties are desired.
8
Our Estimand
• Probabilities of Selection 2)
• The probability of selection is a composite measure of signal, noise, and sample size
9
Our Estimand• Advantages (The Hope) over the concept of p-value
– Close to what many scientists want.
– The intuitive interpretation is correct.
– Does not rely on the correctness of a data generating model for meaning.
– Less ambitious than 3)
• Disadvantages– We changed the question.
– Some may think that there is no need for a confidence interval—wrong.
– Non-regular
10
Estimators
• Why is this a hard problem?– The desire for good frequentist properties – The fact that effect sizes tend to be small relative to
the noise.– This is a non-regular problem—bias is of the same
order as variance.
• Back of the envelope calculations:
11
Estimators
•
• Use plug-in estimator
• Plug-in estimator is 1 – p-value (Goodman, 1992)!– Nonregular
• Near a uniform distribution if
• If n is large, close to 0 or 1 otherwise
– We can expect to be small.
12
Estimators
• Try a Bayesian approach. – Random sample from a , – Flat prior on , known – Use as an estimator of
–
• Bayesian methods do not eliminate non-regularity.
13
Estimators
Focus on MSE in formulating estimators for . 1) Assume is approximately normal with mean
and variance 1) Flat prior (e.g. Killeen’s prep)
2) Normal Prior:
3) Prior is mixture between N(0,1) with probability w point mass on with probability 1-w
14
Estimators
Focus on MSE in formulating estimators for . 2) Single bootstrap (Efron & Tibshirani:1989) .
• This is 1 - p-value. No assumption of approximate normality. If is approximately normal then this is approximately the plug-in estimator:
3) Double bootstrap• This is a bagged plug-in estimator. This bags the 1-
bootstrap p-value. No assumption of approximate normality.
15
Why a double bootstrap?
Double bootstrap estimator for . • Bagging is used to trade variance for bias when
estimators are unstable (Buehlman & Yu, 2002). • The bootstrap estimator of is
unstable; if it does not converge as the sample size increases.
• Under local alternatives such as the bootstrap estimator is inconsistent as well.
16
Double Bootstrap
Double bootstrap estimator for .If has an approximate normal distribution then the
double bootstrap estimator is
That is, the double bootstrap reduces to prep in this case.
17
MSE Plots
• Two groups, each of size 25
• Two distributions (normal, bimodal)
• Two definitions of – –
• Compare – prep, pnorm, pmix, single bootstrap, double
bootstrap
21
Estimators
Instead of a point estimator, consider a confidence interval for .
Assume has an approximate normal distribution; then
In this case a confidence interval for can be found from a confidence interval for the standardized effect size:
22
STAR*D
• Sequenced Treatment Alternatives to Relieve Depression
• Large multi-site study focused on individuals whose depression did not remit with citalopram
• In this trial each individual can proceed through up to 4 stages of treatment. The individual moves to a next stage if the individual is not responding to present treatment.
• Each stage involves a randomization.
23
STAR*D
• This is a data from 683 individuals who did not respond to citalopram and preferred a switch in treatment.
• These individuals were randomized between Venlafaxine, Bupropion, Sertraline
• Outcome: Time until remission.
• We model the area under the survival curve from entry into this stage of treatment until 30 months. (e.g. min(T, 30)).
STAR*D
Regression formula at level 2:
25
STAR*D
• For each s,
• Double Bootstrap– Inner-most bootstrap counts proportion of “votes”
in which – Outer-most bootstrap averages over the proportion
across the bootstrap samples
28
Discussion
• Definition of the probability of selection when there is more than two treatments.
• Confidence intervals for comparisons between more than two treatments.
• Is there a minimax estimator of the selection probability?
• Is there hope for the replication probability?
29
Truth in Advertising:STAR*D
Missing Data + Study Drop-Out
• 1200 subjects begin level 2 (e.g. stage 1)
• 42% study dropout during level 2
• 62% study dropout by 30 weeks.
• Approximately 13% item missingness for important variables observed after the start of the study but prior to dropout.
30
This seminar can be found at:http://www.stat.lsa.umich.edu/~samurphy/
seminars/HopkinsBiostat09.23.09.ppt
Email me with questions or if you would like a copy!
samurphy@umich.edu
31
Our Estimand
• The probability of selection is a composite measure of signal, noise and sample size
• The p-value is a composite measure of estimated signal, estimated noise and sample size.