LT 4.2 Designing Experiments

Thanks to James Jaszczak, American Nicaraguan School

3 Important Principles of Experimental Design

1. Control the effects of lurking variables on the response

2. Randomize by using impersonal chance to assign experimental units to treatments

3. Replicate each treatment on many units to reduce chance variation in the results

Randomized Comparative Experiments

• Randomization produces groups that should be similar in all respects before the treatment is applied

• Comparative design ensures that influences other than the treatment operate equally on both groups

• Therefore, differences in the two groups must be due to the treatments or that random chance favored one group over the other

Control Group• We can reduce the effects of Confounding

variables by using a control group

• It is the first basic principle of statistical design of experiments

• A control group is a group that doesn’t get the treatment

• Comparison of several treatments in the same environment is the simplest form of control

Control Group

• The results of the control group are compared to those of the experimental group

• Since the same confounding factors are present in both groups the only difference to show up should be the effect of the treatment

• Without control groups the confounding effects of things like the placebo effect can take over and dominate the results, making useless treatments seem very effective

Randomization

• Comparison of treatments is valid only if the groups are approximately equal

• We attempt to match groups by elaborate balancing acts

• In medicine we try to match age, sex, weight, blood cholesterol--anything we think might affect the results

The Remedy

• The remedy for bias that inevitably creeps in is Randomization, the second big idea in statistical design

• It doesn’t depend on any characteristic of experimental units or the judgment of the experimenter

• The simplest designs create 2 groups, each randomly selected

• Randomization is an essential ingredient for good experimental design

Replication

• In selecting any two groups there will always be some differences

• Therefore it is impossible to say with just one trial that the treatment is absolutely the cause of the difference

• The more experimental units we assign to the treatment the more those chance variations balance out

• “Use enough EU’s to reduce chance variation” is another big idea in experimental design--Replication

Placebo Effect

• Documented in the early part of the 20th century, the Placebo Effect has confounded medical experiments

• The placebo effect is what happens when the subject feels better after having a treatment

• Patients are often given a placebo, usually a sugar pill, to measure the actual effect of a treatment

Statistical Significance

• We use mathematics to see if the response to a treatment is so large that it could not happen just by chance

• If they are we say they are statistically significant

• Statistical significance is when an effect could occur only rarely just by chance

• Notice that it is not impossible, only very unlikely

Advantages to Experiments

• Experiments can often give good evidence for causation

• Observational studies can only show correlation, not necessarily causation

• Experiments allow us to study specific factors. We change only one thing to study its effect

• Experiments allow us to study the combined effects of several factors simultaneously

Studies

• A Study is when we actually do something and monitor the response

• Experimental Units are the individuals on which the experiment is done

• Subjects are humans that are the experimental units

• Treatments are the specific experimental conditions applied to subjects or experimental units

Explanatory and Response Variables

• A treatment, or explanatory variable, is applied to EU’s to measure the response

• Determining which is the explanatory and response variables becomes very important

• Explanatory variables are sometimes called Factors• Experiments often study the effects of more than

one factor• Each treatment is formed by giving specific Levels

to each of the factors

Completely Randomized

• When all experimental units are allocated at random among all treatments we say the experimental design is Completely Randomized

• Completely randomized experiments can compare any number of treatments

Cautions about Experimentation

• We have to be able to treat all the experimental units identically in every way except the treatment

• Sometimes we achieve this by making a Double Blind experiment

• In a double blind experiment neither the subject nor the researcher knows what treatment the subject is getting

• This removes the effect of hidden clues being given by the researcher

Lack of Realism

• If students know a test doesn’t count it is far more likely they won’t do well than if they know it does.

• Telling them it won’t count then is not a good idea to test which questions are the ones being tested

• In short the treatment doesn’t replicate the real conditions we want to study

• Lack of realism means we can’t extrapolate our results to the whole population, which makes them pretty useless

Matched Pairs Design

• Matching subjects in various ways can improve our results over complete randomization

• Matched pairs compare just two treatments• Two treatments in a pair form a Block• Blocks are chosen randomly as are the

position or timing of the treatments within the block

Block Designs

• Matched pairs are a type of block design• A Block is a group of experimental units that are chosen to be

similar in some way• Treatments within the block are chosen completely randomly• Block designs can have blocks of any size• Blocking allows more precise conclusions• Blocks are formed on the most important unavoidable sources

of variability among the experimental units• Randomization will then average out the differences between

the blocks to give an unbiased comparison of the treatments.