KNR 295 Research Design Slide 1 Design Ch. 6 – Internal Validity.
-
Upload
jeremy-hodges -
Category
Documents
-
view
218 -
download
1
Transcript of KNR 295 Research Design Slide 1 Design Ch. 6 – Internal Validity.
KNR 295Research DesignSlide 1
Design
Ch. 6 – Internal Validity
KNR 295Research DesignSlide 2 Internal Validity
Cause and effect Cause…
“Function: transitive verbInflected Form(s): caused; caus·ingDate: 14th century1 : to serve as a cause or occasion of : MAKE2 : to effect by command, authority, or force- caus·er noun”
*from Merriam-Webster online
KNR 295Research DesignSlide 3 Internal Validity
Cause and effect Effect…
“Function: nounEtymology: Middle English, from Middle French & Latin; Middle French, from Latin effectus, from efficere to bring about, from ex- + facere to make, do -- more at DODate: 14th century1 something that inevitably follows an antecedent (as a cause or agent) ”
*from Merriam-Webster online
KNR 295Research DesignSlide 4 Internal Validity
Cause and effect
Internal validity is a “zero-generalizability concern”
KNR 295Research DesignSlide 5 Internal Validity
Cause and effect So…did the treatment you used
cause the measurement you took to be what it was?
Or…to what extent did the treatment you used influence the measurement you took?
KNR 295Research DesignSlide 6 Internal Validity
Establishing cause and effect Example: Does beer make you
happy? Temporal precedence
beer
happiness
KNR 295Research DesignSlide 7 Internal Validity
Establishing cause and effect Example: Does beer make you
happy? Covariation of cause and effect
If beer then happy
If not beer then not happy
If more beer then more happy
If less beer then less happy
KNR 295Research DesignSlide 8 Internal Validity
Establishing cause and effect Example: Does beer make you
happy? No plausible alternative explanations“Never drink alone”…social
causes?To drink, you need to be flush…it’s an economic difference?
Drinkers experience smoke too…it’s all down to ciggies?
When you drink you go to the loo more often…something to do with bladder swelling/emptying?
Ran
k –
most
p
lau
sib
le?
- least
p
lau
sib
le?
KNR 295Research DesignSlide 9 Internal Validity
Single group threats Sticking with the beer idea…
Imagine you take a group of students, and give them a dose of beer. You might measure their happiness just once at the end, or you might measure them both before and after the beer dose…
KNR 295Research DesignSlide 10 Internal Validity
Single group threats Sticking with the beer idea…
Give beer
Measure happiness
Measure happiness
Measure happiness
Give beer
KNR 295Research DesignSlide 11 Internal Validity
Single group threats Sticking with the beer idea…
History Threat It’s not the beer. It’s some other major historical
event that influenced your group’s happiness… Perhaps on the day of the test, Saddam decided
to pack it in, and everyone simultaneously found out that the recession was a big mass-induced dream state, awaking to find that the world was infinitely happier than they had first thought?
Pla
usi
ble
?
KNR 295Research DesignSlide 12 Internal Validity
Single group threats Sticking with the beer idea…
Maturation Threat As people get older, they get happier. You took
your first measurement during your group’s teenage angst years, then waited until the wisdom, wonder and general delight of 40ish life settled in before taking the second measurement
[can be anything related to general “internal” change in people – any “big event” external to the person causing the change is generally regarded as a history threat]
Pla
usi
ble
?
KNR 295Research DesignSlide 13 Internal Validity
Single group threats Sticking with the beer idea…
Testing Threat (Pre-post only) Measuring happiness made participants focus on
their degree of happiness – if they were unhappy, they resolved to buckle up, and cheer up. If they were happy they were made aware of this and were therefore delighted to be so well-adjusted
Pla
usi
ble
?
KNR 295Research DesignSlide 14 Internal Validity
Single group threats Sticking with the beer idea…
Instrumentation Threat (pre-post again) A couple of possibilities…
The group are scored with different instruments on each occasion. The second is more sensitive than the first. The group’s happiness doesn’t change, but the scores arising from the instruments differ anyway
The person giving out the happiness instrument is such a happy-go-lucky lass that she infects everyone else with her sense of “joi de vivre” (experimenter as instrument)- the longer she’s around the happier you feel
Pla
usi
ble
?
KNR 295Research DesignSlide 15 Internal Validity
Single group threats Sticking with the beer idea…
Mortality Threat As the study wears on and the beer starts to
have its inevitable effect, many people are past the stage of being happy and are approaching that other unfortunate stage of influence…but they aren’t around for the happiness measurement at the end of the studyP
lau
sib
le?
KNR 295Research DesignSlide 16 Internal Validity
Single group threats Sticking with the beer idea…
Regression Threat [The Fugawe Tribe] When you look at the scores of your group on
the pre-test, you notice that they are unusually low… This situation, where for any reason scores are just
by chance suppressed on the first measurement, will probably result in artificially higher scores (in comparison to the first) on the second reading
It’s not that the treatment is having an effect, it’s that the scores are just going back to normal
Normally occurs when selecting a group by pre-test scores…
Pla
usi
ble
?
KNR 295Research DesignSlide 17 Internal Validity
Single group threats Sticking with the beer idea…
Regression Threat You want to find out if the effect even works on
really unhappy people. So you get 200 grad students to fill out a happiness questionnaire, and select only the lowest 25% for your study.
Lo and behold, after a good dose of beer, they all score higher on the hap-o-meter!
Why?
Pla
usi
ble
?
KNR 295Research DesignSlide 18 Internal Validity
Solving single group threats Add a control group The only difference between the
control group and the treatment group should be the presence or absence of the treatment
Sometimes this means you need multiple control groups
Often, it still fails…
KNR 295Research DesignSlide 19 Internal Validity
Multiple group threats All are selection bias threats The critical question is: “were the
groups equal (on the measure of choice) at the start of the study”
The general case is simply a selection bias. Other specific differences follow
KNR 295Research DesignSlide 20 Internal Validity
Multiple group threats
Single-group threats – taken care of now
Multiple-group threats – still a problem
KNR 295Research DesignSlide 21 Internal Validity
Multiple group threats More examples with the hoppy
one… Selection-history threat
Say you got your 2 groups from grad classes On the day of the test, one of the groups’
classes has a nasty test while the other has a normal class
What is causing the difference in happiness now?
KNR 295Research DesignSlide 22 Internal Validity
Multiple group threats More examples with the hoppy
one… Selection-maturation threat
Say you’re looking at the long-term effects of beer on happiness (not temporary as now)
Say also that one group (the drinkers) is selected from among the student population, while the other is selected from among the faculty
It might be easy to imagine that the older group is somewhat more stable with respect to the happiness measure than the younger
KNR 295Research DesignSlide 23 Internal Validity
Multiple group threats More examples with the hoppy
one… Selection-testing threat
Generally, that one group is for some reason more sensitive to be influenced by the first test than the second
Say you get the beer group to volunteer from a pub, while the non-beer group volunteers from a teetotal social club
The pub group also (unknown to you) score higher on a neuroticism scale. This leads them to be more easily influenced by knowledge of their score on a happiness scale…and thus alters their responses second time around
KNR 295Research DesignSlide 24 Internal Validity
Multiple group threats More examples with the hoppy
one… Selection-instrumentation threat
Say we have our happy-go-lucky experimenter from the single group threats operating here too, but she can’t do both groups (schedule can’t allow), so she gets her partner (who is rather gloomy…funny how that seems to happen) to monitor the other group
KNR 295Research DesignSlide 25 Internal Validity
Multiple group threats More examples with the hoppy
one… Selection-mortality threat
Even if you add another (non-beer) group, it is quite probable that your non-beer group will be intact at the end of the study, while your beer group might suffer substantial drop-out
KNR 295Research DesignSlide 26 Internal Validity
Multiple group threats More examples with the hoppy
one… Selection-regression threat
Say you select your volunteers from a pub on a weekday afternoon. The others are selected at random from grad research classes
Trouble is, folk who drink at a pub on a weekday afternoon tend to be a depressed lot…and thus are more likely to improve their scores on a subsequent post-test
[this is fiction, folks!]
KNR 295Research DesignSlide 27 Internal Validity
Solving Multiple group threats Randomly assign to groups Random assignment ensures no
systematic difference between groups
We’re still not done yet though…
KNR 295Research DesignSlide 28 Internal Validity
Social interaction threats
Single-group threats – taken care of by adding control group
Multiple-group threats – taken care of by random assignment
Social interaction threats
KNR 295Research DesignSlide 29 Internal Validity
Social interaction threats One more time with the amber
nectar (as Ben Franklin once said, “beer is proof that God loves us and wants us to be happy”) Diffusion or imitation of treatment
Imagine you pluck all your participants from a college hall of residence. They are aware of each other’s participation, and the other group’s treatment. On seeing the beer group go off for their treatment, the non beer group think “that’s a good idea” and troop off to the liquor store…
KNR 295Research DesignSlide 30 Internal Validity
Social interaction threats One more time…
Compensatory rivalry In some way the groups become aware of each
other’s treatments and of the measures, and something about them sets up a different behavior that influences the final measurement
For example, the non-beer set may for some reason not want to be judged as less happy” than the beer group second time around.
KNR 295Research DesignSlide 31 Internal Validity
Social interaction threats One more time…
Resentful demoralization Say the non-beer group are aware the beer
group is getting the beer Say also the non beer group enjoy a beer They’re jealous This unhappiness finds its way into the post-test
KNR 295Research DesignSlide 32 Internal Validity
Social interaction threats One more time…
Compensatory equalization Say the non-beer group are aware the beer
group is getting the beer Say also the non beer group enjoy a beer Say the experimenters are aware of this They treat the other group with extra kindness
and comfort to offset the perceived slight at the lack of beer
KNR 295Research DesignSlide 33 Internal Validity
Social interaction threats And one extra I’d add here
(repeated from construct validity) Experimenter bias
The belief in the treatment in some way causes the experimenter to behave differently around the two groups, thus turning the findings into a self-fulfilling prophecy
KNR 295Research DesignSlide 34 Introduction to design
KNR 295Research DesignSlide 35 Types of designs
KNR 295Research DesignSlide 36 Types of designs