GROUP-LEVEL DESIGNS

Chapter 9

CHARACTERISTICS OF “IDEAL” EXPERIMENTS

• Research designs that can establish a causal relationship between variables

• Six characteristics1. Time order of the independent variable (IV)

2. IV is manipulated

3. The IV and DV have a relationship

4. Rival hypotheses are controlled for

5. At least one control group is used

6. Random assignment is used

Controlling the Time Order of Variables

• The Independent Variable (IV) must occur before change in the dependent variable (DV) is observed

Manipulating the Independent Variable

• Three ways to manipulate the IV1. One group experiences the IV (X is present)

and one group does not (X is absent)

2. Adjust the “dosage” or amount of exposure of the IV (small amount of X versus large amount of X)

3. One group experiences the IV (X is present) and one group experiences an alternative intervention (something else)

Establishing Relationships between Variables

• The relationship between the independent and the dependent variables must be established in order to infer a cause-effect relationship

Controlling Rival Hypotheses

• Rival hypotheses – a plausible alternative explanation to the research hypothesis

• Three ways to rule out other extraneous variables that might affect the dependent variable

1. Holding Extraneous Variables Constant

2. Using Correlated Variation

3. Using Analysis of Covariance

Holding Extraneous Variables Constant

• Any extraneous variables that might affect the dependent variable are held constant in an ideal experiment by– Being applied equally to all research

participants• If gender is thought to affect the DV, then limit the

sample to only females

– Remaining unchanged for the duration• If dosage of medication is thought to affect the DV, then keep

the dosage constant

Using Correlated Variation

• If several independent variables are used in a research study, then determine to what degree they are correlated– If the correlation between two independent

variables is high, then only one of those variables needs to be included

Using Analysis of Covariance

• A statistical method that is used to compensate for differences between the groups being compared in a research study

Using a Control Group

• The group in the research study that does not receive the intervention (or IV)– A control group is only effective when

research have been randomly assigned to either the experimental or the control group

• Symbols– R = Randomization (selection or assignment)– O = Observation (measurement) of the DV– X = Independent variable or IV

Randomly Assigning Research Participants to Groups

• After research participants have been randomly selected from the population, they are randomly assigned to either an experimental or control group– Research participants are assigned to either

group on the basis of chance (they have an equal chance of being in the experimental or control group)

Matched Pairs

• Another method of dividing research participants into comparison groups– Research participants are matched on key

characteristics, then one individual from each pair is place into two separate groups

• Individuals that do not have a pair are eliminated from the study

INTERNAL AND EXTERNAL VALIDITY

• Group-level designs are evaluated based on their ability to generate knowledge– Internal Validity – the degree to which a

research design can ensure that the independent variable is the sole cause of change in the dependent variable

– External Validity – the extent to which the findings of a research design can generalized to other groups (population) or situations

Threats to Internal Validity (Box 8.1)

• History• Maturation• Testing• Instrumentation Error• Statistical Regression

• Differential Selection• Mortality• Reactive Effects• Interaction Effects• Inter-group Relations

• Known threats that provide alternative explanations (rival hypotheses) for what might bring about change in the dependent variable

Threats to External Validity (Box 8.2)

• Known threats that limit or restrict the degree to which research study results are generalizable– Pretest-Treatment Interaction– Selection-Treatment Interaction– Specificity of Variables– Reactive Effects– Multiple Treatment Interference– Researcher Bias

GROUP RESEARCH DESIGNS

• Group research designs are categorized along the continuum of knowledge– Exploratory Designs– Descriptive Designs– Explanatory Designs

Exploratory Designs

• Do not contain any of the requirements of the “ideal” experiment

• Threats to internal and external validity are high (i.e., virtually all apply)

• These designs are used to explore a research question about which little is already known in order to uncover generalizations and to develop hypotheses for further investigation and testing

One-Group Posttest-Only Design

• Involves a single measure or observation (O1) of the dependent variable that occurs after one group of people has experienced the intervention (X)– Design Blueprint: X O1

• The design does not control for any threats to internal or external validity

Cross-Sectional Survey Design

• Another form of a one-group posttest-only design but the intervention (X) is not specified– A cross-section of a population is observed

(O1) at one particular point in time

– Design Blueprint: O1

Multigroup Posttest-Only Design

• The one-group posttest-only design is applied to multiple groups

• A single measure or observation (O1) of the dependent variable occurs after each group experiences some form of the intervention (X)

• Design Blueprint:

Group 1: X O1

Group 2: X O1

Longitudinal Case Study Design

• Involves repeated measures or observations (O1) of the dependent variable that occurs after one group of people has experienced the intervention (X)– Design Blueprint: X O1 O2 O3

Longitudinal Survey Design

• Involves repeated measures or observations (O1) of one group of people over time– Design Blueprint: O1 O2 O3

• Trend Studies – repeated observations on multiple samples drawn from one population

• Cohort Studies – repeated observations on one group (sample)

Descriptive Designs

• Apply some “Ideal” experiment features:– Time order of variables– Manipulation of the independent variable– Use of comparison group (not a control group)– Random selection but not random assignment

• Compared to exploratory designs, threats to internal and external validity are reduced

Randomized One-Group Posttest-Only Design

• Involves a single measure or observation (O1) of the dependent variable that occurs after one randomly selected group of people has experienced the intervention (X)– Design Blueprint: R X O1

Randomized Cross-Sectional Survey Design

• Another form of a randomized one-group posttest-only design but the intervention (X) is not specified– A randomly selected cross-section of a

population is observed (O1) at one particular point in time

– Design Blueprint: R O1

One Group Pretest-Posttest Design

• Involves repeated measures or observations of the dependent variable– The first observation (O1) occurs before the

intervention (X) and the second observation (O2) occurs after it

• The posttest (O2) is compared to the pretest (O1) to determine if any change occurred in the dependent variable

• Design Blueprint: O1 X O2

Comparison Group Posttest-Only Design

• Involves two groups– The “experimental” group receives the

intervention (X) while the comparison group does not

• A single measure (O1) of the dependent variable occurs for each group

• Design Blueprint: Group 1: X O1

Group 2: O1

Comparison Group Pretest-Posttest Design

• Involves repeated measures or observations of the dependent variable on two groups– An “experimental” and a comparison group

– The first observation (O1) occurs before the experimental group receives X and the second observation (O2) occurs after X

• Design Blueprint: O1 X O2

O1 O2

Interrupted Time-Series Design

• Involves repeated measures or observations of the dependent variable on one group– Several observations occur before the

intervention (X) and several occur after

• Design Blueprint: O1 O2 O3 X O4 O5 O6

Explanatory Designs

• Most closely approximate (and include) the “ideal” experiment

• Most threats to internal and external validity are eliminated or “ruled out”

• The major aim of these designs is to establish a causal connection between interventions (independent variable) and outcomes (dependent variables)

Classical Experimental Design

• Considered to be the “ideal” experiment as all six requirements of are present:– Time order of IV; manipulation of IV;

relationship between IV and DV; rival hypotheses controlled; control group; random selection and random assignment

• Design Blueprint: R O1 X O2

R O1 O2

Randomized Posttest-Only Control Group Design

• Another version of the “ideal” experiment where only one observation or measure is made of the dependent variable for each group

• Design Blueprint: R X O1

R O1

SUMMARY

• Group-level designs are categorized according to the knowledge level continuum– Exploratory, descriptive, explanatory

• Threats to internal and external validity are highest for exploratory designs and lowest for explanatory designs