Experimental Design

Check in

• Exams?– Will be returned on Weds

• Proposal assignment– Posted on the wiki, due 11/9

• Experimental design– Internal validity and threats– In class exercise

Experimental Design

• What is an experiment?– When the researcher manipulates the

independent variable to view change in the dependent variable

• Why do we do experiments?– To establish and study cause and effect

relationships

Famous Experiments

• Alexander Fleming-Discovery of Penicillen– Initially an accident – But he had to experimentally test its influence to

be sure• How would you test if a “mold” killed

bacteria?• How could you make sure something else

didn’t kill the bacteria?

Experimental Design-Example

Experimental Group Zombie Virus Control Group Bacteria

Control Group Zombie Virus

Experimental Design-Example

Experimental Group Zombie Virus Control Group Bacteria

Control Group Zombie Virus

Anti Zombie Serum

Experimental Design-Example

Experimental Group Zombie Virus Dies Control Group Bacteria

Control Group Zombie Virus LIVES!!!

How can we be sure of cause and effect?

• The bacteria died• How do we know it was the serum?

– Not environment (cold, hot, wind)– Not some other contaminant– Not time/age– Not the shape of the dish– Not a preexisting problem with our virus

How can we be sure?

• We control the circumstances• We have a “control group” that is exposed to

everything in exactly the same way…except our intervention

• A true experiment is when the researcher has complete control over who gets the intervention and who does not

Internal Validity

• When we have high control over an experiment, and can confidently dismiss alternate influences, we say that it has “good internal validity”

• Internal validity is the degree to which we can be sure the IV influenced the DV– And we ‘know’ that nothing else caused our

outcome

Internal Validity

• Can be hard to achieve in psychology experiments

• There are many specific “threats” to internal validity

• There are ways to deal with some of these threats

Experimental Design-ExampleThe researcher

Blue Schools Green Schools

The researcher wantsto know if a program will

Prevent early onset sexual activity

Experimental Design-ExampleThe researcher

Blue Schools Green Schools

The researcher wantsto know if a program will

Prevent early onset sexual activity

SEX ED PROGRAM

How will I know?

• If green schools have lower rates of sexual activity, how will I know that its my program?– Can I be sure its not something else?

• What are some threats to knowing that my program is the cause of changes in sexual activity?

This was a real study (is)

• 24 schools middle schools in the greater Boston area

• Randomly assigned by school to – treatment (special sex ed program each year for

three years)– Control (whatever they would normally do)

• Surveyed in 6th, 7th, 8th and 9th grade

Threats to Internal Validity-History

• History– External events or circumstances that influence

outcome– If kids in one or more blue schools had a “pact” to

get pregnant?• More blue kids might get pregnant, and make my green

schools look “better”

– If some green schools gave kids even more sex ed than they are getting from my program?

• This actually happened

How to “control” history?

• Can be quite difficult• Ask participants to avoid other treatments…

– Though this won’t control all exposures– Schools in our study violated this

• Try to collect good, comprehensive data and be aware of the circumstances of your participants as much as possible– At least then you can statistically control for some

influences in your data

Threats…Maturation

• Maturation– When growth may account for our effect– This could certainly happen with a sex program

over a 3 year period

How to control maturation?

• You can’t control it• But you have a control group• Both the experiment and control group should

mature at the same rate– If random assignment has “worked”– And factors related to maturation have been

balanced across conditions• So any difference would reflect your

intervention

Maturation?

• Factors related to sex behavior maturation in teens?– Poverty– Violent neighborhood– Lack of parental monitoring

• Single parenting• Parents who have to work “swing shift”

• So you better hope that random assignment fixed this…

Threats--Selection

• When treatment groups are unequal in a way that influences outcome– If blue schools kids are poorer than green schools?– Poverty increases early onset sexual behavior

• So poorer kids might be harder to “treat”

– Poor kids may have higher rates of sex behavior even before my study starts

– These things could make my treatment look “better”

How to control for selection?

• Randomization– If many poor and wealthy schools, and randomly

assign, groups should be pretty equal• But sample size is crucial here• Stratification may help

• Controlling for Baseline– Measure sexual onset before intervention– Adjust for differences between blue and green

schools– Remaining difference should be due to my

program

Threats-Participant expectations

• When participants believe they are getting a treatment

• So they improve• Also known as placebo effect• E.g. kids know they are getting an education

program• This makes them behave differently

– Not the program itself

Controlling placebo?

• When possible, have a “placebo control”– Control group gets a “treatment”– Gives them attention like intervention group– That should not influence the outcome– That way both treatment and control get

“attention”• Groups “blind” to treatment type

– If possible

Threats—researcher expectations

• When the researcher treats treatment groups differently– Has a different expectation of them– Or actually gives them something extra– Or actually evaluates them differently

• This actually changes the outcome

Researcher expectations

• Can be very subtle• Positive reinforcement to tx group for being

“special”• Giving tx people special information or extra

attention or help

Avoiding researcher expectations?

• Separate the researcher from the treatment• Researcher should be blind to who is in what

treatment– If possible

• Or researcher should be external to treatment process– As in our study

Instrumentation/Measurement

• If your way of measuring changes over time, it could alter how your outcome is measured– E.g. I ask the question “did you have sex”

differently, responses may vary– E.g. If grading papers, and my view of what a

“good answer” changes as I read answers, this could change how I grade later versus earlier responses

Instrumentation?

• Avoid this by piloting your measure first and then sticking with it

• If you must change your measure then collect new data

• Keep track of changes in questions