Educational Research: Educational Research: Experimental StudiesExperimental Studies
EDU 8603
Educational Research
Richard M. Jacobs, OSA, Ph.D.
Research...Research...
The systematic application of a family of methods employed to provide trustworthy information about problems…an ongoing process based on many accumulated
understandings and explanations that, when taken together lead to generalizations about problems and the development of theories
The basic steps of research...The basic steps of research...
Scientific and disciplined inquiry is an orderly process, involving:
description and execution of procedures to collection information (“methodmethod”)
objective data analysisanalysis statement of findings (“resultsresults”)
recognition and identification of a topic to be studied (“problemproblem”)
Research methods...Research methods...
QuantitativeQuantitative……collects and analyzes numerical datadata
obtained from formal instrumentsinstruments
Quantitative methods...Quantitative methods...
descriptive research (“survey research”)correlational researchcausal-comparative research (“ex post facto research”)
experimental research
experimental researchexperimental research…the researcher selects participants and
divides them into two or more groups having similar characteristics and, then, applies the treatment(s) to the groups and measures the effects upon the groups
Conducting an experimental study…Conducting an experimental study…
1. select the problem2. select participants and instrument3. selection and execution of a research plan
4. data analysis and formulation of conclusions
Types of experimental comparison…Types of experimental comparison…
1. comparison of two different approaches (A A versus BB)
2. comparison of an existing approach to a new approach (AA and ~~ A A)
3. comparison of differing amounts of a single approach (AA and a a or aa and AA)
where:
A – experimental (“treatment”) groupB – control (“no treatment,” “nonmanipulated”) group
Variable...Variable...
…a concept (e.g., intelligence, height, aptitude) that can assume any one of a range of values
Research variables...Research variables...
IndependentIndependent……an activity of characteristic believed
to make a difference with respect to some behavior
…(syn.) experimental variable, active variable, cause, treatment
DependentDependent……the change or difference occurring a
a result of the independent variable…(syn.) criterion variable, assigned
variable, effect, outcome, posttest
ConfoundingConfounding……the fact that the effects of the
independent variable may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variable
…(syn.) criterion variable, assigned variable, effect, outcome, posttest
Group experimental designs…Group experimental designs…
1. single-variable2. factorial
single-variable designssingle-variable designs…involve one manipulated
independent variablepre-experimentalpre-experimentalquasi-experimentalquasi-experimentaltrue experimentaltrue experimental
types of pre-experimental designstypes of pre-experimental designs
one-shot case studyone-shot case study
X O
…a single group exposed to a treatment ( X ) and then posttested ( O )
one-group pretest-posttest designone-group pretest-posttest design
O X O
…a single group is pretested ( O ), exposed to a treatment ( X ) and, then, is posttested ( O )
static group comparisonstatic group comparison
X1 O
X2 O…involves at least two groups ( X ), one
receiving a new, or experimental treatment ( X1 ) and another receiving a traditional, or control treatment ( X2 ) and, then, are posttested ( O )
types of quasi-experimental designstypes of quasi-experimental designs
nonequivalent control group nonequivalent control group designdesign
O X O
…random assignment of intact groups that are pretested ( O ), exposed to a treatment ( X ) and then posttested ( O )
time-series designtime-series design
O O O O X O O O O
…a single group is pretested ( O ) repeatedly until pretest scores are stable, exposed to a treatment ( X ) and, then, is repeatedly posttested ( O )
counterbalanced designcounterbalanced design
X1 O X2 O X3 O
X3 O X1 O X2 O
X2 O X3 O X1 O
…all of the groups receive all treatments but in a different order; the number of groups and treatments must be equal
types of true experimental designstypes of true experimental designs
pretest-posttest control group designpretest-posttest control group design
R O X1 O
R O X2 O
…at least two groups are formed by random assignment ( R ), administered a pretest ( O ), receive different treatments ( X1, X2 ), are administered a posttest, and posttest scores are compared to determine effectiveness of treatments
posttest-only control group designposttest-only control group design
R X1 O
R X2 O
…at least two groups are formed by random assignment ( R ), receive different treatments ( X1, X2 ), are administered a posttest, and posttest scores are compared to determine effectiveness of treatments
Solomon four-group designSolomon four-group design
R O X1 O
R O X2 O
R X1 O
R X2 O
…four groups are formed by random assignment ( R ) of participants, two groups are pretested ( O ) and two are not, one pretested and one unpretested group receive the experimental treatments ( X1, X2 ), each group is are administered a posttest on the dependent variable, and posttest scores are compared to determine effectiveness of treatments
factorial designsfactorial designs…involve two or more independent
variables with at least one independent variable being manipulated by the researcher
examples of factorial designsexamples of factorial designs
two-by-two factorial design two-by-two factorial design (four cells)
2 X 2
…two types of factors (e.g., method of instruction) each of which has two levels (e.g., traditional vs. innovative)
A 2 X 2 factorial design…A 2 X 2 factorial design…
Independent Variable
A B
Dep
ende
nt V
aria
ble
O
O
Group #1 Group #2
Group #3 Group #4Cells
not manipulatedmanipulated
A 2 X 2 factorial design…A 2 X 2 factorial design…
A
No interaction between factors
B
A 2 X 2 factorial design…A 2 X 2 factorial design…
ANo interaction between factors
B
A 2 X 2 factorial design…A 2 X 2 factorial design…
A
Interacting factors
B
A 2 X 2 factorial design…A 2 X 2 factorial design…
A
Interacting factors
B
two-by-three factorial design two-by-three factorial design (six cells)(six cells)
2 X 3
…two types of factors (e.g., motivation; interest) each of which has three levels (e.g., high, medium, low)
Single-subject experimental designs…Single-subject experimental designs…
1. A – B – A withdrawal2. multiple baseline designs
3. alternating treatments designs
simple A – B designsimple A – B design…baseline measurements ( O ) are repeatedly
made until stability is established, then the treatment ( X ) is introduced and an appropriate number of measurements ( O ) are made during treatment implementation
simple A – B designsimple A – B design
O O O X O X O X O
baseline treatment phase phase
A | B
A – B – A withdrawal designsA – B – A withdrawal designs…baseline measurements ( O ) are repeatedly made until
stability is established, then the treatment ( X ) is introduced and an appropriate number of measurements ( O ) are made during treatment implementation, followed by an appropriate number of baseline measurements ( O ) to determine stability of treatment ( X )
A – B – A withdrawal designsA – B – A withdrawal designs
O O O X O X O X O O O
baseline treatment baseline phase phase phase
A | B | A
multiple-baseline designsmultiple-baseline designs…used when a return to baseline
conditions is difficult or impossible since treatment effects oftentimes do not disappear when a treatment is removed
…“multiple” refers to the study of more than one behavior, participant, or setting
…instead of collecting baseline data on one specific behavior, data are collected on: (1) several behaviors for one participant, (2) one behavior for several participants, or (3) one behavior and one participant in several settings
…then, over a period of time, the treatment is systematically applied to each behavior (or participant, or setting) one at a time until all behaviors (or participants or settings) have been exposed to the treatment
multiple baseline designmultiple baseline design
Example: one treatment for three behaviors in three settings
behavior 1 O O OXOXOXOXOXOXOXOXOXOXOXO setting #1
behavior 2 O O O O O OXOXOXOXOXOXOXO setting #2
behavior 3 O O O O O O O O OXOXOXO setting #3
A B
the baseline remains the same…
…while the treatment is applied at other
settings
alternating treatments designalternating treatments design…the relatively rapid and random
alternation of treatments ( T1 T2 …Tn ) for a single subject but not necessarily within fixed intervals of time
T1 - T2 - T1 - T1 - T2 - T1 - T1 - T2
The concept of validity…The concept of validity…
…the experiment tests the variable(s) that it purports to test
Threats to validity…Threats to validity…
…internalinternal: factors other than the independent variable that affect the dependent variable
…externalexternal: factors that affect the generalizability of the study to groups and settings beyond those of the experiment
Threats to internal validity…Threats to internal validity…
1. history2. maturation
3. testing
4. instrumentation
5. statistical regression
6. differential selection of participants
7. mortality
8. selection-maturation interaction
historyhistory…the occurrence of events that are not
part of the experimental treatment but that occur during the study and affect the dependent variable
maturationmaturation…the physical, intellectual, and
emotional changes that occur naturally in a study’s participants over a period of time
testingtesting…refers to improved scores on a
posttest as a result of having taken a pretest
instrumentationinstrumentation…the unreliability or lack of
consistency in measuring instruments that can result in an invalid assessment of performance
statistical regressionstatistical regression…the tendency of participants who
score highest on a test to score lower on a second, similar test and vice versa
differential selection of participantsdifferential selection of participants…the outcome when already formed
groups are compared raising the possibility that the groups were different before a study even begins
mortalitymortality…the case in which participants drop
out of a study which changes the characteristics of the groups and may significantly affect the study’s results
selection-maturation interactionselection-maturation interaction…if already-formed groups are used in
a study, one group may profit more (or less) from a treatment or have an initial advantage because of maturation, history, or testing factors
Threats to external validity…Threats to external validity…
1. pretest-treatment interaction2. selection-treatment interaction
3. multiple treatment interference
4. specificity of variables
5. treatment diffusion
6. experimenter effects
7. reactive effects
pretest-treatment interactionpretest-treatment interaction…the situation when participants
respond or react differently to a treatment because they have been pretested
multiple-treatment interferencemultiple-treatment interference…the situation when the same
participants receive more than one treatment in succession
selection-treatment interferenceselection-treatment interference…the situation when participants are
not randomly selected for treatments
specificity of variablesspecificity of variables…the situation when a study is conducted with (1) a
specific kind of participant; (2) is based on a particular operational definition of the independent variable; (3) uses specific dependent variables; (4) transpires at a specific time; and, (5) under a specific set of circumstances
treatment diffusiontreatment diffusion…the situation when different
treatment groups communicate with and learn from each other
experimenter effectsexperimenter effects…the situation when the researchers
present potential threats to the external validity of their own studies
reactive arrangementsreactive arrangements…the situation when a number of
factors associated with the way in which a study is conducted interacts with or shapes the feelings and attitudes of the participants involved
Types of reactive arrangements…Types of reactive arrangements…
…Hawthorne effectHawthorne effect: any situation in which participants’ behavior is affected not by the treatment per se but by their knowledge of participating in a study
…compensatory rivalrycompensatory rivalry: the control group is informed that they will be the control group for a new, experimental study (“John Henry effectJohn Henry effect”)
…placebo effectplacebo effect: the situation in which half of the participants receive no treatment but believe they are
…novelty effectnovelty effect: the situation in which participant interest, motivation, or engagement increases simply because they are doing something different
Controlling for extraneous Controlling for extraneous (confounding) variables…(confounding) variables…
1. randomization2. matching3. comparing homogeneous groups or subgroups
4. using participants as their own controls
5. analysis of covariance (ANCOVA)
randomizationrandomization…the process of selecting and assigning
participants in such a way that all individuals in the defined population have an equal and independent chance of being selected for the sample
matchingmatching…a technique for equating groups on
one or more variables, usually the ones highly related to performance on the dependent variable (e.g., pairwise matching)
comparing homogeneous groups or subgroupscomparing homogeneous groups or subgroups
…a technique to control an extraneous variable by comparing groups that are similar with respect to that variable (e.g., stratified sampling)
using participants as their own controlsusing participants as their own controls…exposing a single group to different
treatments one treatment at a time
analysis of covariance (ANCOVA)analysis of covariance (ANCOVA)…a statistical method for equating
randomly formed groups on one or more variables by adjusting scores on a dependent variable for initial differences on some other variable
Data analysis and interpretation…Data analysis and interpretation…
for single-subject research…a visual inspection and analysis of
graphical presentations of results…focuses upon: adequacy of the design;
an assessment of treatment effectiveness (clinicalclinical vs. statistical significancestatistical significance)
Mini-Quiz…Mini-Quiz…
True and false…
…experimental researcher is the only type of research that can test hypotheses to establish cause-and-effect relationships
True
True and false…
…the manipulation of the independent variable is the one characteristic that differentiates correlational research from other types of research
False
True and false…
…an experimental study is guided by at least one hypothesis that states an expected causal relationship between two variables
True
True and false…
…the “experiment” in an experimental study is conducted to conform or disconfirm the research problem
False
True and false…
…experimental research has both random selection and assignment whereas causal-comparative research has random selection not assignment
True
True and false…
…it is correct to state that a control group always receives no treatment in experimental studies
False
True and false…
…in an experimental study, the researcher makes every effort to ensure that the two groups start as equivalently as possible on all variables except the independent variable
True
True and false…
…in an experimental study, the researcher collects data on the independent variable from the groups to determine whether there is a significant difference between their performances
False
True and false…
…in an experimental study, the researcher endeavors to make the groups as similar as possible so that the only major difference between them is the treatment variable(s) manipulated by the researcher
True
True and false…
…even though experimental research is the only type of research that can truly establish cause-effect relationships, it is not universally appropriate for all research problems or studies
True
True and false…
…an experiment is valid if results obtained are due only to the control variable and if they are generalizable to individuals or contexts beyond the experimental setting
False
True and false…
…internal validity focuses upon threats or rival explanations that influence the outcomes of an experimental study but are not part of the independent variable
True
True and false…
…if a study’s results cannot be replicated in other settings by other researchers, then the study has low external reliability
False
True and false…
…to conduct a valid experiment, all the researcher has to do is to maximize internal and external validity
False
True and false…
…true experimental designs control for all sources of internal and external validity
False
True and false…
…the more narrowed and controlled a research situation is, the less realistic and generalizable it becomes
True
True and false…
…a study that is not internally valid is worthless
True
True and false…
…the longer a study lasts, the more likely it is that history will be a threat to the study’s internal validity
True
True and false……the way a researcher controls for selection-
maturation interaction is to select a design that controls for potential problems or to make every effort to determine if selection-maturation interaction is operating in the study
True
True and false…
…pretest-treatment interaction makes a study’s results generalizable only to other pretested groups
True
True and false…
…the seriousness of pretest-treatment interaction threat is dependent upon the research participants, the nature of the independent and dependent variables, and the duration of the study
True
True and false…
…even if intact groups are randomly selected, the possibility exists that the experimental group is in some important way different from the control group and/or from the larger population
True
True and false…
…selection-treatment interaction is an uncontrolled variable in research designs involving randomization
True
True and false…
…it is possible to know what a researcher means by the terms assigned to variables without clear operationalized descriptions
False
True and false…
…good designs control for many sources of invalidity, poor designs control few
True
True and false…
…ANCOVA is used to “correct” or adjust posttest scores for initial pretest differences
True
True and false…
…ANCOVA is assumes that the relationship between the independent and covariate variables is curvilinear
False
True and false…
…a researcher should not totally accept or reject a research design because of how it controls (or fails to control) for sources of invalidity
True
True and false…
…a researcher should decide which research design is most appropriate not only by the controls provided but also by the nature of the study and the setting in which it is to be conducted
True
True and false…
…in a static-group comparison design, each group serves as a control or comparison group for the other group(s)
True
True and false…
…in a counterbalanced design, the average performance of the groups on each treatment are calculated and compared
True
True and false…
…in a true experimental designs, participants should be randomly selected or randomly assigned
False
True and false…
…in the pretest-posttest control group design, if groups are essentially the same on the dependent variable at the start of a study, a t-test can be used for analysis; if not, ANCOVA is used
True
True and false…
…the more dissimilar the intact groups in a study are, the stronger the study is
False
True and false…
…when describing and symbolizing factorial designs, the control variable is placed first
False
True and false…
…the purpose of a factorial design is to determine whether the effects of an independent variable are generalizable across all levels or are specific to particular variables
True
True and false…
…factorial designs are very effective for testing research hypotheses that cannot be tested with a single-variable design
True
True and false…
…for all types of research, the more a study’s results are replicated, the less confidence there is in the procedures that produced those results
False
Fill in the blank…
…the two main ways that researchers equate groups participating in experimental research
simple random sampling
stratified random sampling
Fill in the blank…
…the researcher decides what treatment(s) will constitute the independent variable and which group will get which treatment
manipulation
Fill in the blank…
…the researcher’s efforts to remove the influence of any extraneous variable that might affect scores on the dependent variable
control
Fill in the blank…
…the researcher’s concern with threats or factors other than the independent variable that affect the dependent variable
internal validity
Fill in the blank…
…the researcher’s concern with the extent to which the study results can be generalized to groups and settings beyond those of the experiment
external validity
Fill in the blank…
…a threat to validity involving the tendency for scores to more toward a average or expected score
statistical regression
Fill in the blank…
…a threat to internal validity that may be overcome by testing for initial equivalence between groups
differential selection of participants
Fill in the blank…
…the population from which a researcher endeavors to extract a sample
target population
Fill in the blank…
…the actual population from which a researcher extracts a sample
accessible population
Fill in the blank…
…the influences on the part of the experimenter over which the experimenter has no direct control but may affect study procedures
passive experimenter effects
experimenter personal-attributes effects
Fill in the blank…
…the situation in which a researcher's expectation of the study affect the researcher’s behavior and the researcher outcomes
active experimenter effect
experimenter bias effect
Fill in the blank…
…an external variable that effects the independent variable and intertwines with other extraneous variables such that it is difficult to determine the unique effects of each
confounding variable
Fill in the blank…
…the term referring to a design that has more than one independent variable
factor(ial)
Fill in the blank…
…a study to compare the achievement of two groups: one group taught in an anxiety-producing environment and one group taught in an anxiety- reducing environment
experimental
Fill in the blank…
…the nonmanipulated variable in a factorial design
control variable
Fill in the blank…
…a second study conducted by a different investigator, with different participants, behaviors, and settings
systematic replication
Fill in the blank…
…additional studies that add confidence to the procedures that produced those results
replication
Fill in the blank……a second study involving the development of treatment
packages, composed of two or more interventions that have been found to be effective individually, designed for persons with complex behavior disorders
clinical replication
Fill in the blank…
…a second study conducted on a number of participants with the same problem, location, or time
simultaneous replication
This module has focused on...This module has focused on...
experimental studies
...which test hypotheses to establish cause-and-effect relationships
The next module will focus on...The next module will focus on...
descriptive statistics
…the set of mathematical procedures for describing, synthesizing, analyzing, and interpreting quantitative data
Top Related