INTERNAL VALIDITY AND BASIC RESEARCH DESIGN
Internal Validity
the approximate truth about inferences regarding cause-effect or causal relationships
only relevant in studies that try to establish a causal relationship Or wherever the language of the study infers a causal
relationship
Internal Validity
internal validity is a “zero-generalizability concern”
Establishing Cause and Effect
3 Criteria for a causal relationship to exist: temporal precedence covariation of cause and effect no plausible alternative explanations
Example: Does drinking beer make you happy?
Establishing Cause and Effect
1. temporal precedence beer comes before happiness
2. covariation of cause and effect if beer then happy, if no beer then not happy the more beer then the more happy
Establishing Cause and Effect
3. no plausible alternative explanations
“Never drink alone”…social causes?
To drink, you need to have cash…it’s an economic difference?
Drinkers experience smoke too…it’s all about the ciggies?
When you drink you go to the loo more often…something to do with bladder swelling/emptying?a
re t
hese
pla
usi
ble
?
Identifying Possible Threats
Possible studies testing whether beer makes you happy?
give beer
measure happiness
XX OOgive beer
measure happiness
XX OO
measure happiness
OO
KNR 164
KNR 164
give beer
measure happiness
XX OO
measure happiness
OOKNR 164 (8am)
OOOOKNR 164 (9am)
Internal Validity Threats
single-group threats
multiple-group threats
social interaction threats
Single Group Threats
history threat some event (or series of events) occurring during the
study that affects the outcome maturation threat
observed effect due to normal maturation or development of subjects
Single Group Threats
testing threat (pre-post only) taking the pre-test affects performance on post-test
instrumentation threat (pre-post only) change in tests (observations, measures) is
responsible for change in outcome
Single Group Threats
mortality threat non-trivial or non-random dropout affects the results
regression threat it is not the treatment causing the effect, but rather
the posttest scores are simply moving back toward the population mean (up or down)
Solving Single Group Threats
Add a control group the only difference between the control group and the
treatment group should be the presence or absence of the treatment
sometimes this means you need multiple control groups
Multiple Group Threats
all are selection bias threats the critical question is: “were the groups equal (on
the measure of choice) at the start of the study?”
Internal Validity Threats
single-group threats
multiple-group threats
social interaction threats
Multiple Group Threats
selection-history threat some event (or series of events) occurring between
the pre- and post-test affects the groups differently selection-maturation threat
there are differential rates of normal development between the pre-test and post-test for the groups
Multiple Group Threats
selection-testing threat the effect of taking the pre-test varies between groups
selection-instrumentation threat changes in the tests across time vary between the
groups
Multiple Group Threats
selection-mortality threat differential non-random dropout occurs between the
groups, so that post-test scores differ selection-regression threat
differential rates of regression to the mean cause post-test scores to differ across groups.
Solving Multiple Group Threats
randomly assign to groups random assignment ensures no systematic difference
between groups (and therefore decreases the possibility that any observed affect is due to selection bias)
there have to be a reasonable number of participants in each group for the random assignment to work can’t just randomly assign two people to two groups and
expect them to be equal!
Internal Validity Threats
single-group threats
multiple-group threats
social interaction threats
Social Interaction Threats
diffusion or imitation of treatment control group learns about treatment somehow and
does it on their own control group becomes more like treatment group
compensatory rivalry control group knows about treatment and becomes
competitive with the experimental group, which affects how they respond on the posttest
control group becomes more like treatment group
Social Interaction Threats
resentful demoralization basically the opposite of compensatory rivalry upon finding out about the treatment the control
group members become discouraged, angry, etc. probably perform worse on post-test than they would
have compensatory equalization of treatment
the researcher somehow affects the outcome by treating the control group favorably in some way
Social Interaction Threats
experimenter bias the belief in the treatment in some way causes the
experimenter to behave differently around the two groups, thus turning the findings into a self-fulfilling prophecy
Reducing Social Interaction Threats
1. Do the researchers claim in any way that the relationships they uncovered in their study are casual in nature (e.g., the intervention caused changes in the dependent variable, the treatment lead to group differences in the dependent variable)?
2. If so, do you think that the causal inference made by the researchers is reasonable or justifiable given the design of the study? [This is the key Internal Validity question]
The following questions should help answer Question #2. 1. Consider the type of design (e.g. are there multiple groups, how were subjects
assigned to groups, is there a pre and post-test).2. Does their design provide evidence of temporal precedence between the
cause and effect (i.e. did the cause happen before the effect)?3. Does their design provide evidence of covariation between the cause and
effect (i.e. is there an effect when the cause is present, but not when it is not)?4. Are there plausible alternative explanations for the relationship between the
cause and effect? If so, what are they (use the potential single group, multiple group, and or social interaction threats to internal validity as a guide)?
1. Do the researchers claim in any way that the relationships they uncovered in their study are casual in nature (e.g., the intervention caused changes in the dependent variable, the treatment lead to group differences in the dependent variable)?
2. If so, do you think that the causal inference made by the researchers is reasonable or justifiable given the design of the study? [This is the key Internal Validity question]
The following questions should help answer Question #2. 1. Consider the type of design (e.g. are there multiple groups, how were subjects
assigned to groups, is there a pre and post-test).2. Does their design provide evidence of temporal precedence between the
cause and effect (i.e. did the cause happen before the effect)?3. Does their design provide evidence of covariation between the cause and
effect (i.e. is there an effect when the cause is present, but not when it is not)?4. Are there plausible alternative explanations for the relationship between the
cause and effect? If so, what are they (use the potential single group, multiple group, and or social interaction threats to internal validity as a guide)?
Guiding Questions for Critiquing the Internal Validity of Research
Practice
identify potential internal validity issues (using the threats) for each of the following studies
1.The purpose of the study was to examine the relationship between exercise participation and happiness. A total of 300 students served as subjects, all of whom were recruited from a personal fitness class at a small liberal arts school in the mid-western portion of the United States. Students were led through a 30 minute traditional step aerobics class during one of their regularly scheduled class periods. At the end of the class, the students completed the 5-item Happiness subscale from the General Mood Inventory. Results indicated that the students were extremely happy, thus supporting the positive effect of exercise on happiness.
1.The purpose of the study was to examine the relationship between exercise participation and happiness. A total of 300 students served as subjects, all of whom were recruited from a personal fitness class at a small liberal arts school in the mid-western portion of the United States. Students were led through a 30 minute traditional step aerobics class during one of their regularly scheduled class periods. At the end of the class, the students completed the 5-item Happiness subscale from the General Mood Inventory. Results indicated that the students were extremely happy, thus supporting the positive effect of exercise on happiness.
2. A teacher wanted to determine whether allowing students to choose the topics covered in introduction to exercise science class would translate into better understanding of the subject. At the beginning of the semester students in both sections of KNR 164 were given a 100-item exam assessing general knowledge about exercise science. During the course of the semester, students who signed up for Section 1 of the class (MWF 8am) were given a choice by the teacher about which topics they would like to cover during the class period. Students who signed up for Section 2 of the class (MWF 6pm) followed a standard curriculum. At the end of the semester, students in both sections were given different versions of a 200-item exam covering key concepts of exercise science. Results indicated that the students in the experimental class (i.e., those who were allowed to choose the topics) scored significantly higher on the end of the semester exam, suggesting that the type of teaching style affected learning.
3. The purpose of the study was to test whether a mental imagery training package resulted in better athletic performance. Freshman football players from ISU were randomly assigned to a control and experimental group. During a private session at the beginning of the season the players assigned to the experimental group were given 2 hours of instruction on using mental imagery, and were encouraged to use the technique for 10 minutes before each practice and game during the season. The players assigned to the control group also had a private meeting at the beginning of the season where they watched the movie “Remember the Titans”. At the end of the season, the amount of playing time and the head coach’s rating of overall performance for each athlete were compared to see if the imagery training was effective. Athletes who received the imagery training were found to perform better than those who did not receive the training.
3. The purpose of the study was to test whether a mental imagery training package resulted in better athletic performance. Freshman football players from ISU were randomly assigned to a control and experimental group. During a private session at the beginning of the season the players assigned to the experimental group were given 2 hours of instruction on using mental imagery, and were encouraged to use the technique for 10 minutes before each practice and game during the season. The players assigned to the control group also had a private meeting at the beginning of the season where they watched the movie “Remember the Titans”. At the end of the season, the amount of playing time and the head coach’s rating of overall performance for each athlete were compared to see if the imagery training was effective. Athletes who received the imagery training were found to perform better than those who did not receive the training.
Top Related