Experiments & Observational Studies

Chapter 13

We’ve developped a new rabbit food, HIPPITY HOP.

Hippity Hoprabbit food

Makes fur soft and shiny!

Increases energy!

100% of daily vitamins &

essential oils!

Experimental unit -

The single individual (person, animal, plant, etc.) to which the different treatments are assigned

Factor -

The explanatory variable(what you’re changing for each subject)

Level -

A specific value for the factor(each specific option for a factor)

For our Hippity Hop example:

- the rabbits

- the type of rabbit food we give to each rabbit

- probably 2 levels: Hippity Hop, and some other type of rabbit food that we’re going to compare Hippity Hop to in this experiment

Treatment -

A specific experimental condition applied to the unitsmade up of factors

at different levels

Response Variable -

what you are measuring at the end of an experiment(after the treatments have been applied)

You must be specific!!!!

- In the rabbit food example, we have 2 treatments

- hair softness and shine?amount of energy?

Example 1: A farm-product manufacturer wants to determine if the yield of a crop is different when the soil is treated with three different types of fertilizers.Fifteen similar plots of land are planted with the same type of seed but are fertilized differently. At the end of the growing season, the mean yield from the sample plots is compared.

Experimental Units?

Factors?

Levels?

How many treatments?

Response variable?

plots of land

type of fertilizer

fertilizer types A, B, C

3

yield of crop

Example 2: A consumer group wants to test cake pans to see which works the best (bakes evenly). It will test

aluminum, glass, and silicone pans in both gas and electric ovens.

Experimental Units?

Factors?

Levels?

How many treatments?

Response variable?

cake batter

Two factors - Type of pan & Type of oven

Type of pan has 3 levels (aluminum, glass, silicone), and type of oven has

2 levels (electric, gas)

6

how evenly the cake bakes

Experimental DesignIn 2007, deaths of a large number of pet dogs and cats were ultimately traced to contamination of some brands of pet food. The manufacturer has checked and tested the food, and now claims the food is safe, but before it can be released to the general public, an experiment to test whether the food is now truly safe for dogs and cats to eat must be conducted.

What would the treatments be?

New food from the company, and another that we are certain is safe

What would the response variable be?

Health of the pets, as assessed by a veterinarian

Experimental Design

A group of 32 dog owners have volunteered their pets for this experimental study. Explain how you would carry out a completely* randomized experiment to see if the new food is safe for dogs to eat. The dogs will eat the assigned food for a period of 6 weeks.

*completely randomized experiment means NO BLOCKING ALLOWED!! I’ll talk about

what this means later

When explaining an experimental design, you must: 1. Describe how you will randomly assign your subjects to

the different treatment groups.2. Say what each of the treatment groups will do.3. Explain what response variable you will

evaluate/compare at the end of the experiment.

Remember to … BE SPECIFIC!!!

COMPLETELY RANDOMIZED DESIGN

Group of 32 dog

volunteers

Group 1: 16 dogs

Group 2: 16 dogs

Treatment 1: dogs eat new

food for 6 weeks

Treatment 2: dogs eat “safe”

food for 6 weeks

Compare health of dogs, to be

evaluated by a veterinarian

Completely randomized experiment means NO BLOCKING ALLOWED!

Randomassignment

(explaining the randomization procedure)● Number the dogs with a unique number from 1 to 32.● Use a RNG on a calculator/computer to pick 16 UNIQUE random numbers

from 1 to 32 (ignore any repeats).● The dogs whose numbers are chosen will be placed in treatment group 1.

The remaining dogs will be placed in treatment group 2.

OR

● Using a RNG on calculator/computer, RANDOMLY assign each dog a unique number from 1 to 32.

● The dogs with numbers 1-16 are placed in treatment group 1. The rest of the dogs will be placed in treatment group 2.

You MUST explain the randomness procedure - even if you use a diagram to design/describe the experiment!

You may also describe the experiment all in paragraph form, which may look something like this...Give each of the 32 dogs a unique number from 1 to 32. Then use a random number generator to select 16 unique, random numbers from that range.

The 16 dogs whose numbers are selected will be assigned to Treatment Group 1, and will eat the new dog food for 6 weeks. The remaining 16 dogs will be assigned to Treatment Group 2, and will eat the “safe” dog food for 6 weeks.

After the 6 weeks are over, we will compare the health of the dogs, as assessed by a veterinarian.

Why is randomization

important?

In theory, randomization...

● ‘equalizes’ all other variables● gives us treatment groups

that are more or less the same

-BUT-

● Variation WILL occur!!Sometimes we end up with ‘unfair’ groups. :(

This is why we replicate the experiment (perform the experiment a number of times, with different groups of subjects, in different

locations, etc).

Principles of Experimental Design1. Control

Make conditions as similar as possible for all treatment groups (aside from the actual treatments, of course).

If we observe a difference between groups, we want to be able to say that difference is because of the treatments, and not something else!

2. Randomization

The use of chance to assign subjects/units to treatments

3. Replication

Repeat the experiment on many subjects/in different locations/etc.

Replication is also helpful because, in most experiments, we’re dealing with a group of volunteers, NOT a random sample.

Replication helps us apply our results to a larger population.

BLOCKING! (A form of control)

Of the 32 dogs, 12 are poodles, and 20 are German Shepherds.Explain the changes you would make to your previous design by incorporating blocking.

BLOCKING! (A form of control)

Of the 32 dogs, 12 are poodles, and 20 are German Shepherds.Explain the changes you would make to your previous design by incorporating blocking.

“We will block by breed of dog (separate poodles from German Shepherds, before we start assignment of treatments), because ….

BE SURE TO EXPLAIN WHY!!(think back to our reasoning for taking a stratified random sample - it’s similar reasoning here!)

Group of 32 dog

volunteers

12 poodles

20 German Shepherds

Block by breed!

Group 1: 6 poodles

Group 2: 6 poodles

Group 3: 10 German Shepherds

Group 4: 10 German Shepherds

Random!

Random!

Treatment 1: dogs eat new food for 6 wks

Treatment 2: dogs eat ‘safe’ food for 6 wks

Treatment 1: dogs eat new food for 6 wks

Treatment 2: dogs eat ‘safe’ food for 6 wks

Compare health of poodles

Compare health of German

Shepherds

YIKES!Plus, remember we also need to explain our randomness procedure for assigning subjects to treatment groups!

Blocking is to experiments as__________________ is to sampling.stratifying

Is the difference I observed in my results…“statistically significant?”

What does that even mean?!?

statisticallysignificant

When an observed effect is

SO LARGEthat it would rarely occur by chance

we call that effect ‘statistically significant.’

SAT PREP CLASSES!● Higher SAT scores in ONLY 6 weeks!● top-secret far-eastern study strategies!● 3 sessions a week, only 5 hours per session!● All the other kids are being forced into it by their

parents, so why not?!● COST: only $30,000!! What a bargain!● Scores will improve, GUARANTEED, or your

money back!

SAT PREP CLASSES!

Group of 20

students

take this class

mean score improvement of …

18 points!!!It worked!!!

CONTROL GROUP

a group that is used to compare the factor against

Can be a placebo, or the “old”/current item

SAT PREP CLASSES!

Group of 20

students

Group A: These kids care about their

grades, so they choose to take this SAT prep class Group A scored

WAYYYYYYYYhigher!

IT WORKED!!Group B:

These kids DON’T CARE about their grades, so they DON’T

take the prep class

ConfoundingTwo variables are CONFOUNDED when they BOTH have an association with the response variable.

(And we cannot tell which of the two variables is actually responsible for the effect.

Or maybe it’s both? We can’t tell!!)

Same idea as confounding …

But a lurking variable is something that we, the researchers, did not

consider before we began the experiment.

And it ended up having an important effect on the response.

(Usually this means … whoops, let’s start over and do some blocking…)

Lurking Variable

ConfoundingTwo variables are CONFOUNDED when they BOTH have an association with the response variable.

(And we cannot tell which of the two variables is actually responsible for the effect.

Or maybe it’s both? We can’t tell!!)

Same idea as confounding …

But a lurking variable is something that we, the researchers, did not

consider before we began the experiment.

And it ended up having an important effect on the response.

(Usually this means … whoops, let’s start over and do some blocking…)

Lurking Variable(an example of possible confounding)A professor at Cornell wants to know the effect of his teaching style, so he hands

out student evaluations for his students to fill out at the end of the semester.

Fall and Spring terms - a total of 472 students

His teaching was identical, exceptsubdued in the fall… enthusiastic in the spring!

Average response?Fall - 2.93

Spring - 4.05

Could the weather have been confounding?

According to Newsweek“Of all pre-college curricula, the highest level of mathematics one studies in secondary school has the strongest continuing influence on bachelor’s degree completion. Finishing a course beyond the level of Algebra 2 (for example, trigonometry or pre-calculus*) more than doubles the odds that a student who enters postsecondary education will complete a bachelor’s Degree.”

Propose a study design that might enable us to draw this conclusion. Why would it be difficult (and probably unethical) to perform this study as an experiment? Can we randomly assign students to take certain math classes, or to NOT

take certain math classes? Are there confounding variables that affect a student’s success in college,

aside from what math classes they take in high school?

about control...

testing a new headache medicine...

Group of 40

volunteers

Group 1: 20 patientsTake the pill we’re

testingCompare

number ofheadaches Group 2: 20 patients

NO PILL Sigh… Look at me, I didn’t get “the good

stuff.” I’m such a loser, I think I’ll go get a

headache now.

PLACEBOa “dummy” treatment that has no effect on the response variable

(well … we say ‘no effect’ but there is such a thing as the ‘placebo effect’)

Basically, sometimes people actually have

physical results simply because they’re

getting SOME kind of treatment. Even if it’s

not the real thing, people THINK it is … and that’s somehow

enough?

testing a new headache medicine...

Group of 40

volunteers

Group 1: 20 patientsTake the pill we’re

testingCompare

number ofheadaches Group 2: 20 patients

Take a placebo (fake pill)

BLINDINGsubjects do not know which treatment they are getting

DOUBLE-BLINDneither the subjects, nor the evaluator* knows which treatment each subject received

*the evaluator - means whoever checks the subject for improvement, but this also includes the person who administers the treatment, or cares for the subject during the experiment - basically, anybody who has direct contact with the subject

We can accomplish blind/double-blind experiments by making sure that our treatments

look, smell, feel, taste exactly the same (or as much as possible).

A pharmaceutical company is testing a new headache medicine pill, and wishes to test it in an experimental study against a control group.

Is it possible for the experiment to be blinded? double blinded?

Sure! Just make sure the real pill and the placebo pill both look, taste, feel, etc. exactly the same!

A cutting-edge treatment for combatting brain tumors involves drilling tiny holes in the patient’s skull and injecting a drug cocktail into the brain.

Is it possible for the experiment to be blinded? double blinded?

Is it ethical?Uh, yeah, we’ll just drill tiny holes in the control group patients’ brains

and inject … water? I’m sure everybody will be cool with that.

LAST EXAMPLE!(I promise)

Example of a study on stomach ulcers...In 1958, an initial experiment of a very creative treatment for stomach ulcers involved:

● Anesthetizing the patient● Putting a balloon in the patient’s stomach● Filling the balloon with freezing coolant

In this initial experiment, there were 24 patients. All were cured.

This treatment became very popular, even more so than surgery, but there

were some SKEPTICS …. Could this be the result of placebo effect?

New experiment, 1963A controlled experiment was performed.

A group of 160 ulcer patients were divided into 2 groups -

82 received the gastric freezing treatment - 34% improved

78 received a placebo treatment - 38% improved

So … Gastric freezing was NO better

than a placebo!

We NEED comparison in experiments - otherwise it’s really hard to tell when a treatment is truly working, or when other factors are causing the change.