Post on 25-Feb-2016
description
Experiments:Validity, Reliability and Other Design Considerations
At least three different meanings:
▪ Controlled Studies▪ Control in an
Experiment▪ Controls used in a
study or analysis
2
Assigning people to groups vs. observing people who ‘assign’ themselves
Example of pitfalls in experimental assignment:
▪ Portacaval Shunt Example
Examples of key pitfalls of observational studies:
▪ Cervical cancer and circumcision study
▪ Alcohol consumption and lung cancer
3
Measurement Concerns Construct Validity
Design Concerns Internal Validity External Validity Ecological Validity
4
“Face” validity deals with subjective judgement of appropriate operationalization
“Content” validity is a more direct check against relevant content domain for the given construct.
5
How do we know that our independent variable is reflecting the intended causal construct and nothing else?
6
Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.
7
Effect
? ?
? ?
History ▪ additional I.V. that occurs
between pre-test and post-test
Maturation▪ Subjects simply get older and
change during experiment
Testing▪ Subjects “get used” to being
tested
Regression to the Mean▪ Issue with studies of extremes on
some variable8
Demand Characteristics▪ Anything in the experiment that could guide subjects to expected outcome
Experimenter Expectancy▪ Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)
9
Evaluation Apprehension
“Hawthorne Effect” Temporary improvement based on
observation
Solutions Double-blind experiments Experiments in natural setting (i.e.,
subjects do not know they are in an experiment)
Cover stories Hidden measurements
10
Naïve experimenter Those conducting study are not aware of theory or
hypotheses in the experiment Blind
Researcher is unaware of the experiment condition that he/she is administering
Standardization Experimenter follows a script, and only the script
“Canned” Experimenter Audio/Video/Print material gives instructions
11
Selection Bias▪ Issue with non-random selection of subjects
Mortality▪ Departure of subjects in the experiment
Diffusion, Sharing of Treatments▪ Control group unexpectedly obtains treatment
Other ‘social’ threats?▪ Compensatory rivalry, resentful demoralization, etc.
12
Setting
Population
History
13
External Validity– How far does the given experiment generalize to similar groups, individuals, etc?
14
15
Internal Validity
External & Ecological Validity
Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.
Some experiments can be conducted in a real-world setting while maintaining random assignment and manipulation of treatments
16
Milliman (1986) Study of music tempo and restaurant customer behavior
Cheshire and Antin (2008) Study of Incentives and Contributions of Information in an Online Setting
1998 Total Solar Eclipse: testing temperature of sun’s corona
17
Pro’s Gives researcher tight control over independent factors Allows researcher to test key relationships with as few
confounding factors as possible Allows for direct causal testing
Con’s Often very small N; enough for statistical purposes but
not ideal for generalizability Sometimes give up large amounts of external validity in
favor of construct validity and direct causal analysis Require a large amount of planning, training, and time–
sometimes to test relationship between only 2 factors!
18
Cost and Effort Is the effort worth it to test the concepts you are
interested in?Manipulation and Control
Will you actually be able to manipulate the key concept(s)?
Importance of GeneralizabilityAre you testing theory, or trying to establish a
real-world test?
19