Designing Experiments

Section 4.2 Part 1 & 2

Reference Text:

The Practice of Statistics, Fourth Edition.

Starnes, Yates, Moore

Starter• In late 1995, a Gallup survey reported that Americans approved

sending troops to Bosnia by 46 to 40 percent. The poll did not mention that 20,000 U.S. troops were committed to go. A CBS News poll mentioned the 20,000 number and got the opposite outcome – a 58 to 33 percent disapproval rate. Briefly explain why the mention of the number of troops would cause such a big difference in the poll results. Write the name of the kind of bias that is at work here.

• A church group interested in promoting volunteerism in a community chooses an SRS of 200 community addresses and sends members to visit these addresses during weekday working hours and inquire about the residents’ attitude toward volunteer work. Sixty percent of all respondents say that they would be willing to donate at least an hour a week to some volunteer organization. Bias is present in this sample design. Identify the type of bias involved and state whether you think the sample proportion obtained is higher or lower than the true population proportion.

Objectives• Describe the difference between an observational study and

an experiment.• Observational:

– Lurking Variables -Confounding

• Experiment:– Treatment -Experimental units -Subjects

• Bad Experiments vs good experiments – Random assignment -Control Group– Complete Randomized Design

• Principals of Experimental design– 1, 2, 3 important things to have!

• What can go wrong?!

Observational vs. Experimentstudy

• An observational study surveys individuals to measure variables of interest. No attempt is made to influence the response.– The goal of an observational stud can be to describe some

group or situation, to compare groups, or to examine relationships between variables.

• An experiment deliberately imposes a treatment on individuals and observes the response.– The goal of an experiment is to determine whether the treatment

causes a change in the response

– The treatment imposed is the explanatory variable– The response observed is the response variable

Observational Study

• An observational study, even one based on a random sample, is a poor way to gauge the effect that changes in one variable have on another variables.

• To see the response variable change, we must actually impose the change.

• When our goal is to understand cause and effect, experiments are the only source of fully convincing data.

Lurking VariablesConfounding

• We already know about lurking variables…– Makes it hard to see the true relationship between the

explanatory and response variable.

• If Lurking variables start to become intertwined with the response variables where we don’t know which variable is responsible then it becomes a confounding variable…

• Confounding: Occurs when two variables are associated in such a way that their effects on a response variable cannot be distinguished from each other.

• Example pg 231 & 232 - Hormones

Check for Understanding

• Does reducing screen brightness increase battery life in lap top computers? To find out, researchers obtained 30 new laptops of the same brand. They chose 15 of the computers at random and adjusted their screens to the brightness setting. The other 15 laptop screens were left at a default setting- moderate brightness. Researchers then measured how long each machine’s battery lasted.

• Was this an observational study or an experiment? Justify your answer.

Check for Understanding

• Does eating dinner with their families improve students’ academic performance? According to an ABC News article “teenagers who eat with their families at least 5 times a week are more likely to get better grades in school” this finding was based on a sample survey conducted by researchers at Columbia University

• Was this an observational study or an experiment? Justify your answer.

• What are the explanatory and response variable?• Explain clearly why such a study cannot establish a cause-and-

effect relationship. Suggest a lurking variable that may be confounded with whether families eat dinner together.

The language of ExperimentsRemember: Experiments can give good evidence of

causation• Treatment- a specific condition applied to the

individuals in an experiment.

– This is what we actually do to “them”• “them” can be people, animals, or objects

• Experimental units- are the smallest collection in individuals to which treatments are applied. When units are human beings, they are often called subjects.

• Factor - another name for an explanatory variable (x)

• Remember: Experiments can give good evidence of causation

Example:• TB pg 234: A Louse-Y-Situation• A study published in the New England Journal of Medicine

(March 11,2010) compared two medicines to treat head lice: an oral medication called ivermectin and a topical lotion containing malathion. Researchers studied 812 people in 376 households in seven areas around the world. Of the 185 households randomly assigned to ivermectin, 171 were free from head lice after two weeks compared with only 151 of the 191 households randomly assigned to malathion.

• Identify:– The experimental units– Explanatory and response variables– Treatments in this experiment

How to experiment Badly • We can take a look at how to experiment badly by

looking at some examples:• TB Pg 236 “Does caffeine affect pulse rate?”• Many students regularly consume caffeine to help

them stay alert. Thus, it seems plausible that taking caffeine might increase an individual’s pulse rate. Is this true? One way to investigate this is to have volunteers measure their pulse rates, drink some cola with caffeine, measure their pulses again after 10 minutes, and calculate the increase in pulse rate.

• …….

TB Pg 236 “Does caffeine affect pulse rate?”

• Unfortunately, even if every student’s pulse rate went up, we couldn’t attribute the increase to caffeine. Perhaps the excitement of being in an experiment made their pulse rates increase. Perhaps it was the sugar in the cola and not the caffeine. Perhaps their teacher told them a funny joke during the 10-minute waiting period and made everyone laugh! In other words, there are many variables that are potentially confounding with taking caffeine.

How to experiment well

• If treatements are given to groups that differ greatly when the experiment begins, bias will result.

• Allowing personal choice will bias our results in the same way that volunteers bias the results in online opinion polls.

• Solution to reduce bias: Random assignment…

Random Assignment

• In an experiment, random assignment means that experimental units are assigned to treatments at random, that is, using some sort of chance process.

• Completely Randomized Design: The treatments are assigned to all the experimental units completely by chance.– To avoid AP exam error (points loss) defer to

the “Hat Method”

Hat Method• 50 students agreed to participate in an experiment.

We want a completely random design. Describe how you randomly assign 25 students to each of the two treatments ( A and B).

• Answer: Write each subject’s name on one of the slips of paper. Put all slips in the hat and mix them thoroughly. Draw them out one at a time until you have 25 slips. These 25 will participate in A. The other 25 will participate in B.

Outline for Complete Random Design

RandomAssignment

Group 125 students

Group 225 students

ImposeTreatment A

ImposeTreatment B

CompareResults

50 students

Control Group

• The primary purpose of a control group is to provide a baseline for comparing the effects of the other treatments.

• Necessary? Some experiments don’t include control groups. That’s appropriate if researchers simply want to compare the effects of several treatments, and not to determine whether any of the work better than an inactive treatment.

Summary: Principles of Experimental Design: hand out

• The basic principles of designing experiments are as follows:

1. Control for lurking variables that might affect the response. Use comparative design and ensure that the only systematic difference between the groups is the treatment administered.

2. Random Assignment: Use impersonal chance to assign experimental units to treatments. This helps create roughly equivalent groups of experimental units by balancing the effect of lurking variables that aren’t controlled on the treatment groups.

3. Replication: Use enough experimental units in each group so that any differences in the effect of the treatment can be distinguished from chance differences between groups.

Break?

What Can Go Wrong?!

• Placebo Effect: A response to a dummy treatment.– Example: A good illustration of the placebo

effect is when a parent kisses a child’s “boo-boo” when the child gets injured. Even though the kiss has no “active treatment,” it makes the child feel better!

Do Placebos Work?

• Want to help balding men keep their hair? Give them a placebo- one study found that 42% of balding men maintained or increased the amount of hair on their heads when they took a placebo.

• The strength of the placebo is hard to pin down because it depends on the exact environment, but “placebos work” is a good place to start when you think about planning medical experiments.

Example

• You want to test a new drug to see if it provides pain relief to migraine sufferers.

• Draw a schematic diagram of an experimental design that could be used on a group of 30 subjects.

Experiment to test a new drug using 30 chronic migraine sufferers as subjects

30 Subjects:Choose SRS

of 15

TreatmentGroup

Chosen 15

PlaceboGroup

Other 15

Give drug

Giveplacebo

CompareResults

Double-Blind• In a double blind experiment, neither the subjects nor

those who interact with them and measure the response variable know which treatment a subject is received.

• Balding example: its foolish to tell the subject they’re getting a placebo, but its also foolish for the doctor to know that because they might expect to see less than if the subject was receiving the actual treatment.

• So both administrator and subject don’t know which is which. Placebo or treatment. Double-blind

Statistically significant

• An observed effect so large that it would rarely occur by chance is called statistically significant.

• If we observe statistically significant differences among the groups in a randomized comparative experiment, we have good evidence that the treatment actually caused these differences!

• We will learn much more about what it means to be statistically significant in Ch 9-12

• “Unlikely to Happen by Chance”

Statistically Significant: Causation

• We know that in general a strong association does not imply causation. A statistically significant association in data from a well-designed experiment does imply causation.

• “Unlikely to Happen by Chance”• Remember: Experiments can give good evidence of

causation

Objectives• Describe the difference between an observational study and

an experiment.• Observational:

– Lurking Variables -Confounding

• Experiment:– Treatment -Experimental units -Subjects

• Bad Experiments vs good experiments – Random assignment -Control Group– Complete Randomized Design

• Principals of Experimental design– 1, 2, 3 important things to have!

• What can go wrong?!

Homework