Experimental Design

Experimental Design

• Experimental design is the part of statistics that happens before you carry out an experiment

• Proper planning can save many headaches

• You should design your experiments with a particular statistical test in mind

Why do experiments?

• Contrast: observational study vs. experiments

• Example: – Observational studies show a positive

association between ice cream sales and levels of violent crime

– What does this mean?

Why do experiments?

• Contrast: observational study vs. experiments

• Example: – Observational studies show a positive

association between ice cream sales and levels of violent crime

– What does this mean?

Alternative explanation

Hot weather

Ice cream

Violentcrime

Alternative explanation

Hot weather

Ice cream

Violentcrime

Correlation is not causation

Why do experiments?

• Observational studies are prone to confounding variables: Variables that mask or distort the association between measured variables in a study– Example: hot weather

• In an experiment, you can use random assignments of treatments to individuals to avoid confounding variables

Goals of Experimental Design

• Avoid experimental artifacts• Eliminate bias

1. Use a simultaneous control group2. Randomization3. Blinding

• Reduce sampling error1. Replication2. Balance3. Blocking

Goals of Experimental Design

• Avoid experimental artifacts• Eliminate bias

1. Use a simultaneous control group2. Randomization3. Blinding

• Reduce sampling error1. Replication2. Balance3. Blocking

Experimental Artifacts

• Experimental artifacts: a bias in a measurement produced by unintended consequences of experimental procedures

• Conduct your experiments under as natural of conditions as possible to avoid artifacts

Goals of Experimental Design

• Avoid experimental artifacts• Eliminate bias

1. Use a simultaneous control group2. Randomization3. Blinding

• Reduce sampling error1. Replication2. Balance3. Blocking

Control Group

• A control group is a group of subjects left untreated for the treatment of interest but otherwise experiencing the same conditions as the treated subjects

• Example: one group of patients is given an inert placebo

The Placebo Effect

• Patients treated with placebos, including sugar pills, often report improvement

• Example: up to 40% of patients with chronic back pain report improvement when treated with a placebo

• Even “sham surgeries” can have a positive effect

• This is why you need a control group!

Randomization

• Randomization is the random assignment of treatments to units in an experimental study

• Breaks the association between potential confounding variables and the explanatory variables

Experimental unitsC

onfo

undi

ng v

aria

ble

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

Without randomization, the confounding variable differs among treatments

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

With randomization, the confounding variable does not differ among treatments

Blinding

• Blinding is the concealment of information from the participants and/or researchers about which subjects are receiving which treatments

• Single blind: subjects are unaware of treatments

• Double blind: subjects and researchers are unaware of treatments

Goals of Experimental Design

• Avoid experimental artifacts• Eliminate bias

1. Use a simultaneous control group2. Randomization3. Blinding

• Reduce sampling error1. Replication2. Balance3. Blocking

Replication

• Experimental unit: the individual unit to which treatments are assigned

Experiment 1

Experiment 2

Experiment 3

Tank 1 Tank 2

All separate tanks

Replication

• Experimental unit: the individual unit to which treatments are assigned

Experiment 1

Experiment 2

Experiment 3

Tank 1 Tank 2

All separate tanks

2 Experimental Units

2 Experimental Units

8 Experimental Units

Replication

• Experimental unit: the individual unit to which treatments are assigned

Experiment 1

Experiment 2

Experiment 3

Tank 1 Tank 2

All separate tanks

2 Experimental Units

2 Experimental Units

8 Experimental Units

Pseudoreplication

Why is pseudoreplication bad?

• problem with confounding and replication!

• Imagine that something strange happened, by chance, to tank 2 but not to tank 1

• Example: light burns out

• All four lizards in tank 2 would be smaller

• You might then think that the difference was due to the treatment, but it’s actually just random chance

Experiment 2

Tank 1 Tank 2

Why is replication good?

• Consider the formula for standard error of the mean:

SEY

s

n

Larger n Smaller SE

Balance

• In a balanced experimental design, all treatments have equal sample size

Better than

Balanced Unbalanced

Balance

• In a balanced experimental design, all treatments have equal sample size

• This maximizes power

• Also makes tests more robust to violating assumptions

Blocking

• Blocking is the grouping of experimental units that have similar properties

• Within each block, treatments are randomly assigned to experimental treatments

• Randomized block design

Randomized Block Design

Randomized Block Design

• Example: tanks in a field

Very sunny

Not So Sunny

Block 1

Block 4

Block 2

Block 3

What good is blocking?

• Blocking allows you to remove extraneous variation from the data

• Like replicating the whole experiment multiple times, once in each block

• Paired design is an example of blocking

Experiments with 2 Factors

• Factorial design – investigates all treatment combinations of two or more variables

• Factorial design allows us to test for interactions between treatment variables

Factorial Design

5.5 6.5 7.5

25 n=2 n=2 n=2

30 n=2 n=2 n=2

35 n=2 n=2 n=2

40 n=2 n=2 n=2

Tem

pera

ture

pH

Interaction Effects

• An interaction between two (or more) explanatory variables means that the effect of one variable depends upon the state of the other variable

Interpretations of 2-way ANOVA Terms

0

10

20

30

40

50

60

70

25 30 35 40

Temperature

Gro

wth

Rate

pH 5.5

pH 6.5

pH 7.5

Effect of pH and Temperature,No interaction

0

5

10

15

20

25

30

35

40

45

25 30 35 40

Temperature

Gro

wth

Rate

pH 5.5

pH 6.5

pH 7.5

Interpretations of 2-way ANOVA Terms

Effect of pH and Temperature,with interaction

Goals of Experimental Design

• Avoid experimental artifacts• Eliminate bias

1. Use a simultaneous control group2. Randomization3. Blinding

• Reduce sampling error1. Replication2. Balance3. Blocking

What if you can’t do experiments?

• Sometimes you can’t do experiments

• One strategy:– Matching– Every individual in the treatment group is

matched to a control individual having the same or closely similar values for known confounding variables

What if you can’t do experiments?

• Example: Do species on islands change their body size compared to species in mainland habitats?

• For each island species, identify a closely related species living on a nearby mainland area

Power Analysis

• Before carrying out an experiment you must choose a sample size

• Too small: no chance to detect treatment effect

• Too large: too expensive

• We can use power analysis to choose our sample size