Sample size determinationSample size determination

Nick Barrowman, PhDSenior Statistician

Clinical Research Unit, CHEO Research Institute

March 29, 2010

OutlineOutline

• Example: lowering blood pressure• Introduction to some statistical

issues in sample size determination• Two simple approximate formulas• Descriptions of sample size

calculations from the literature

ExampleExample

• Physicians design an intervention to reduce blood pressure in patients with high blood pressure

• But does it work? Need a study.• How many participants are required?• Too few: may not detect an effect

even if there is one.• Too many: may unnecessarily expose

patients to risk.

The null hypothesisThe null hypothesis

• For intervention studies, the null hypothesis is usually this: on average there is no effect.

• “Innocent until proven guilty”• The physicians who designed the

intervention believe the null hypothesis is false.

• The study is designed to test the null hypothesis.

• Often write H0 for the null hypothesis.

The studyThe study

• The population is considered to be all people who might be eligible for the intervention (might depend on age, other medical conditions, etc.)

• Study participants are viewed as a sample from this population.

• Suppose for each study participant we measure blood pressure at baseline, and after 6 weeks of intervention

• Outcome is change in blood pressure

• H0 is that mean change in BP is 0.

Population Population vs.vs. sample sample

Population

Population mean of the change in blood pressure

Random sample

Inference Sample mean of the change in blood pressure

Calculation

Population distribution of change in blood pressure

mean

± 1 standard deviation

Probability distributionsProbability distributions

Recall that variance is the square of the standard deviation, often written as

Population distribution of change in blood pressurePopulation distribution of change in blood pressure

Sampling distributionSampling distribution of of meanmean change in blood change in blood pressure (N=1)pressure (N=1)

Sampling distributionSampling distribution of of meanmean change in blood change in blood pressure (N=2)pressure (N=2)

Sampling distributionSampling distribution of of meanmean change in blood change in blood pressure (N=5)pressure (N=5)

Sampling distributionSampling distribution of of meanmean change in blood change in blood pressure (N=10)pressure (N=10)

Increasing sample size reduces the variability of the sample mean.

standard deviation

standard error

SD

NSE =

Variance and sample sizeVariance and sample size

• As we’ve seen, increasing the sample size is akin to reducing the variance

• Equivalently, reducing the variance (e.g. using a more precise measurement device) can reduce the sample size requirements

Hypothesis testHypothesis test

Sampling distribution of the mean under the null hypothesis,a.k.a. the null distribution

Hypothesis testHypothesis test

Rejection region

Observed mean

Reject the null hypothesis if the observed mean is far in the tails of the null distribution, i.e. we have ruled out chance

Possible scenariosPossible scenarios

Based on the study findings we infer either …

that the intervention has no effect (accept H0)

that the intervention has an effect (reject H0)

or

Possible scenariosPossible scenarios

In reality, either …the intervention has no effect (H0 is true)

the intervention has an effect (H0 is false)

Based on the study findings we infer either …

that the intervention has no effect (accept H0)

that the intervention has an effect (reject H0)

or

or

FourFour possible scenarios possible scenarios

In reality, either …the intervention has no effect (H0 is true)

the intervention has an effect (H0 is false)

Based on the study findings we infer either …

that the intervention has no effect (accept H0)

that the intervention has an effect (reject H0)

or

or

Four possible scenariosFour possible scenarios

In reality, either …the intervention has no effect (H0 is true)

the intervention has an effect (H0 is false)

Based on the study findings we infer either …

that the intervention has no effect (accept H0)

Correctly accept H0

that the intervention has an effect (reject H0)

or

or

Four possible scenariosFour possible scenarios

In reality, either …the intervention has no effect (H0 is true)

the intervention has an effect (H0 is false)

Based on the study findings we infer either …

that the intervention has no effect (accept H0)

Correctly accept H0

that the intervention has an effect (reject H0)

Correctly reject H0

or

or

Four possible scenariosFour possible scenarios

In reality, either …the intervention has no effect (H0 is true)

the intervention has an effect (H0 is false)

Based on the study findings we infer either …

that the intervention has no effect (accept H0)

Correctly accept H0

that the intervention has an effect (reject H0)

Type-I error Correctly reject H0

or

or

Four possible scenariosFour possible scenarios

In reality, either …the intervention has no effect (H0 is true)

the intervention has an effect (H0 is false)

Based on the study findings we infer either …

that the intervention has no effect (accept H0)

Correctly accept H0 Type-II error

that the intervention has an effect (reject H0)

Type-I error Correctly reject H0

or

or

Type-I errorType-I error

If the null hypothesis is true, the rejection region of the test represents type-I error.

The probability of type-I error is the area of the red region below, and is denoted by .

Type-II errorType-II error• Type-II error is failing to reject the null

hypothesis when it is false.• The probability of type-II error is denoted

.• It depends on how big the true effect is• Sample size calculations require

specification of an alternative hypothesis, which indicates the size of effect we would like to detect

Type-II errorType-II error

Type-II errorType-II error

Type-II errorType-II error

Relationship between type-I and type-II errorRelationship between type-I and type-II error(alpha=0.05)(alpha=0.05)

Relationship between type-I and type-II errorRelationship between type-I and type-II error(alpha=0.10)(alpha=0.10)

Relationship between type-I and type-II errorRelationship between type-I and type-II error(alpha=0.20)(alpha=0.20)

Relationship between type-I and type-II errorRelationship between type-I and type-II error

• Sample size calculations depend on the tradeoff between type-I and type-II error.

• We usually fix the probability of type-I error (alpha) at 5% and then try to minimize the probability of type-II error (beta).

• Define Power = 1 – beta• We want to maximize power• One way to do this is by increasing the

sample size

How sample size affects powerHow sample size affects power

Sample size (doubled)Sample size (doubled)

Sample size (quintupled)Sample size (quintupled)

An approximate formula for the blood An approximate formula for the blood pressure examplepressure example

• Suppose the variance in the change in blood pressure, sigma2, is the same for the null and alternative hypotheses

• Suppose alpha is fixed at 0.05 and we use two-sided tests (allowing for the possibility that blood pressure could be either increased or decreased by the intervention)

• Then we will have approximately 80% power to detect a mean change in blood pressure delta if we enroll N participants, where

N = 8 sigma2 / delta2 (approximately)

ExampleExample

• Suppose the standard deviation of the change in blood pressure is anticipated to be 7 mmHg (so the variance is 49)

• Suppose we fix alpha at 0.05 and we’d like to have approximately 80% power to detect a mean change of 5 mmHg

• Then we would need about 16 participants

When there are two groupsWhen there are two groups

• So far, the example has used a single group of study participants

• Usually we want to compare two groups: a control group that receives “standard of care” or placebo, and an experimental group that receives a new intervention

• This is how most randomized controlled trials are set up

• In this case, delta is the difference between the means of the two groups.

• For simplicity, assume that the variance is the same in the two groups.

An approximate sample size formula An approximate sample size formula for the case of two groupsfor the case of two groups

• A similar approximate formula applies, again assuming alpha=0.05 and power=80%:

N per group = 16 sigma2 / delta2

(approximately)

• Careful! This is the required sample size per group.

• Also, note that the constant is double what is was for the case of a single group.

• So the total sample size is 4 times as large.

ExampleExample

• Suppose we want to compare patients randomized to placebo with patients randomized to a new intervention

• Suppose the standard deviation is anticipated to again be 7 mmHg (so the variance is 49)

• Suppose we fix alpha at 0.05 and we’d like to have approximately 80% power to detect a change of 5 mmHg

• Then we would need about 32 participants per group, for a total of about 64 participants

SummarySummary

• increases with variance• decreases with size of effect to detect• decreases with probability of type-I error,

alpha• decreases with probability of type-II error,

beta

Required sample size …

Sample size determination has many Sample size determination has many other aspectsother aspects

• Different types of outcomes: dichotomous (e.g. mortality), time-to-event (e.g. survival time), etc.

• Different designs: observational studies (e.g. case-control), surveys, prevalence studies

• Practical considerations: e.g. costs, feasibility of recruitment

Questions?Questions?

αα = Probability of type-I error = Probability of type-I error

(Rejecting the null hypothesis when it is in fact true.)

Power = 1 Power = 1 –– ββ

(Rejecting the null hypothesis when it is in fact false.)

Review: A comedy of errors …Review: A comedy of errors …

Probability of a false conviction

Probability of a true conviction