Lally School of Management & Technology Michael J. Kalsher PSYCHOMETRICS MGMT 6971 1 MGMT 6971...
-
Upload
georgia-sanders -
Category
Documents
-
view
213 -
download
0
Transcript of Lally School of Management & Technology Michael J. Kalsher PSYCHOMETRICS MGMT 6971 1 MGMT 6971...
Lally School of Management &
Technology
Michael J. Kalsher
PSYCHOMETRICSMGMT 6971
1MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Week 1: Introduction and Research Design
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Course Overview
• Review of research design/methodology and statistical concepts
• Review of SPSS (data entry; setting up variables; graphing; syntax; etc.)
• Statistical analysis techniques– Covariance, correlation, simple regression, multiple regression– t-tests, ANOVA / ANCOVA / MANOVA– Non-parametric statistics– Factor analysis, Multilevel Linear Models, Structural Equation
Models
• Grading requirements– Exams, Labs, Problem Sets, Data Collection/Analysis Project
2
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Research Methods & Design: Establishing Control over your variables
• Historical foundations of scientific research in the behavioral and social sciences.
• The importance of research design– Ruling out alternative explanations.– Establishing control of IVs.
• Research Design vs. Statistical Analysis
3
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Methods of Establishing Truth• Tenacity
– “It’s so because it’s so”• Authority
– “Aristotle said it’s so”• Logical Deduction (Rationalism)
– Aristotle said women have fewer teeth than men (Premise)– You are a woman– Therefore, you have fewer teeth than I
• Empiricism– Combines Logical Deduction with observation
(measurement)– “Let’s count your teeth”
4
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Scientific Method
• Shared observations– Rules out individual experiences like religious
revelations or esthetic experiences (William James).
• Reproducible Effects– “No miracles”
• Conditional Truths– Premises may be wrong– Necessary Connection may be wrong
5
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Types of Relationship (between two concepts)
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Spurious Relationships
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Spurious RelationshipsIce Cream Sales
Swimming Pool Drownings
Heat Wave
A city's ice cream sales are found to be highest when the rate of drownings in the city’s swimming pools is highest. To allege that ice cream sales cause drowning, or vice-versa, would be to imply a spurious relationship between the two. In reality, a third variable, in this instance a heat wave, more likely caused both.
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Sets of Relationships (a theory)
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
High Experimental Research
Differential Research
Correlational Research
Case-study Research
Low Naturalistic Observation
Exploratory Research
De m
and
Research plan becomes increasingly detailed (e.g., precise hypotheses and analyses) but less flexible.
Research plan may be general, ideas, questions, and procedures relatively unrefined.
A Model of the Research Process: Levels of Constraint
(Model used to illustrate the continuum of demands placed on the adequacy of the information used in research and on the nature of the processing of that information.)
10
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Independent variable A variable that is actively manipulated by the researcher to see what its impact
will be on other variables.
Dependent variable A variable that is hypothesized to be affected by the independent-variable
manipulation.
Extraneous variable Any variable (usually unplanned or uncontrolled factors), other than the
independent variable, that might affect the dependent measure in a
study.
A constant Any variable prevented from varying (by holding variables constant, they do
not affect the outcome of the research).
Classes of Research Variables:
Variables defined by their use in research
11
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Variable values are represented by numbers, but these numbers may not demonstrate all the characteristics of true numbers.
1. Nominal. A variable made up of discrete, unordered categories. Each category is either present or absent and categories are mutually exclusive and exhaustive (e.g., gender).
2. Ordinal. A variable for which different values indicate a difference in the relative amount of the characteristic being measured.
3. Interval. A variable for which equal intervals between variable values indicate equal differences in amount of the characteristic being measured.
4. Ratio. Ratios between measurements as well as intervals are meaningful because there is a starting point (zero).
Classes of Research Variables:
The Measurement Model
12
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Levels of Measurement
Nominal Ordinal Interval Ratio
Diagnostic categories Socioeconomic Test scores; Weight; length;
brand names; political class; ranks personality and reaction time;
or religious affiliation attitude scales # of responses
Identity Identity; magnitude Identity; magnitude Identity; magnitude;
equal intervals equal intervals;
true zero point
None Rank order Add; subtract Add; subtract;
multiply; divide
Nominal Ordered Score Score
Chi Square Mann-Whitney t-test; ANOVA t-test; ANOVA
U-test
Examples
Properties
MathematicalOperations
Type of Data
Typical Statistics
Scales of Measurement: Some Examples
13
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
The Role of Variance
14
- In an experiment, IV(s) are manipulated to cause variation between experimental and control conditions.
- Experimental design helps control extraneous variation--the variance due to factors other than the manipulated variable(s).
Sources of Variance- Systematic between-subjects variance
Experimental variance due to manipulation of the IV(s) [The Good Stuff]
Extraneous variance due to confounding variables.
Natural variability due to sampling error
- Non-systematic within-groups varianceError variance due to chance factors (individual differences) that affect some participants more than others within a group
[The Not-So-Good Stuff]
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Separating Out The Variance
15
SST = Sums of Squares Total
SSM = Sums of Squares Model
SSR = Sums of Squares Error
SST
SSM SSR
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Controlling Variance in Experiments
16
In experimentation, each study is designed to:
1. Maximize experimental variance.
2. Control extraneous variance.
3. Minimize error variance.• Good measurement• Manipulated and Statistical control
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Controlling Variance in Observational Studies
17
• Choose IV’s with large natural variance• Control for alternate explanations by
measuring confounding variables and statistically removing their variance
• Minimize error variance– Good measurement– Statistical control
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Maximizing Experimental Variance: Strong manipulations and Manipulation Checks
18
Experimental Variance (The Good Stuff)
Due to the effects of the IV(s) on the DV(s)
Ensure that experimental manipulations are strong and reliable!
Manipulation CheckProcedures designed to determine whether manipulation
of the IV(s) had the intended effect(s) on the DV(s)
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Controlling Extraneous Variance
19
Extraneous variables: Between-group variables--other than the IV(s)--that have effects on whole groups and thus may confound the results.
Goal: To prevent extraneous variables from differentially affecting the groups.
Solution: Take steps to ensure that: (1) the experimental and control groups are equivalent at the beginning of the study; and (2) groups are treated exactly the same--save for the intended manipulation (of the IV).
Methods (for controlling extraneous variance): 1. Random Assignment of subjects to experimental conditions2. Select participants on the basis of one or more potentially confounding
variables (e.g., age, ethnicity, social class, IQ, sex). 3. Build the confounding variables into the study as additional IVs.4. Match participants on confounding variable or use within-subjects design
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Test Statistics
20
Essentially, most test statistics are of the following form:
Test statistic = Systematic variance
Unsystematic variance
Test statistics are used to estimate the likelihood that an observed difference is real (not due to chance), and is usually accompanied by a “p” value (e.g., p<.05, p<.01, etc.)
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
A Very Simple Statistical Model
outcomei = (model) + errori
• model – an equation made up of variables and parameters
• variables – measurements from our research (X)
• parameters – estimates based on our data (b)
outcomei = (bXi) + errori
outcomei = (b1X1i + b2X2i + b3X3i)+ errori
21
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Examples of Statistical Models
• One Predictor (e.g. deviance):outcomei = (bXi) + errori
outcomelecturer1 = mean + errorlecturer1
errrorlecturer1 = mean – outcomelecturer1 = 1 – 2.6 = -1.6
• Multiple Predictors (e.g. sum of squared errors):outcomei = (b1X1i + b2X2i…)+ errori
errori = (outcome1 – model1)2 + (outcome2 – model2)2 …
= (-1.6)2 + (-0.6)2 + (0.4)2 + (1.4)2 = 5.20
22
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Types of Hypothesis
• Neyman and Pearson proposed organizing scientific statements into testable hypotheses.– H0 – null hypothesis, that no effect will occur
• Adding a narrative component to a video game will not affect gameplay experience
– H1 – alternative (or experimental) hypothesis, that the effect you are testing for will occur
• Playing a game with a narrative component will improve your gameplay experience
• Data cannot prove alternative hypotheses, only reject null ones 23
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Null Hypothesis Significance Testing (NHST)
• NHST combines Fisher’s work with Neyman and Pearson’s– Initially assume null hypothesis is true– Choose a statistical model that represents an
alternative hypothesis– Calculate p-value of the null hypothesis producing
this model– If p < .05 (generally), model fits and alternative
hypothesis is supported
• We’re never certain, we just have evidence24
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
One- and Two-tailed Tests
• One-tailed: directional results (effect is present or not)
• Two-tailed: directional results (effect increases, decreases, or no effect)
25
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Types of Mistakes
26
Statistical decision
Reject Ho
Don’t reject Ho
True state of null hypothesis
Ho true Ho false
Type I error Correct
Correct Type II error
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Inflated Error Rates
• A measure of how well Type I errors have been avoided
• In most research, the complexity of the question requires more than one test. The rate of error increases with the number of tests done, increasing the Type I error. This is called familywise error.
• Solution? Choose a stricter p-value for each individual test (Bonferroni correction)
required p-value per test = (desired overall p-value)/(number of tests)
27
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Statistical Power
• A measure of how well Type II errors have been avoided (i.e. how well a test is able to find an effect)
• = 1 – type II error rate• Power should be 0.8 or higher, so Type
II error rate should not exceed .20.
28
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Confidence Intervals & Statistical Significance
• p-value of H0 decreases with the amount of overlap between two confidence intervals
• Moderate overlap (defined as ½ the average Margin Of Error) indicates p = .05.
• MOE = ½ the length of the confidence interval:
• So moderate overlap is:
(
29
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Sample Size & Statistical Significance
• Because MOE is a result of sample size (via the confidence interval), small differences can be significant in large samples, and large differences might not be significant in small samples.– This is because larger samples have more
power to detect effects when they exist.
30
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Effect Sizes: The Correlation coefficient
31
The statistical test only tells us whether it is safe to conclude that the means come from different populations. It doesn’t tell us anything about how strong these differences are. So, we need a standard metric to gauge the strength of the effects.
The correlation coefficient (r) is one metric for gauging effect size.
• Ranges from 0 – 1 (no effect to perfect effect)• Rough cutoffs (nonlinear, that is twice the r value
doesn’t necessarily mean twice the effect)– 0.10 – small effect (explains 1% of the variance)– 0.30 – medium effect (explains 9% of the variance)– 0.50 – large effect (explains 25% of the variance)
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Effect Sizes: The coefficient of determination
32
The statistical test only tells us whether it is safe to conclude that the means come from different populations. It doesn’t tell us anything about how strong these differences are. So, we need a standard metric to gauge the strength of the effects.
r2 (r-Square), or the “Coefficient of Determination”, is one metric for gauging effect size.
Rules of Thumb regarding effects sizes:
Small effect: 1-3% of the total variance
Medium effect: 10% of the total variance
Large effect: 25% of the variance
r2 =SSM
SST
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
– Uses the same unit for all data (standard deviation units)
– Provides information about the signal-to-noise ratio – how large is the effect in comparison to other effects on the same data?
– = (the difference of the means) divided by the standard deviation
– Effect cutoffs (but remember this is only rough):• 0.2 – small• 0.5 – medium• 0.8 – large
33
Effect Sizes: Cohen’s d
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Meta-Analysis
• An average of the effect size of multiple studies that all address the same question– Weighted to favor more precise studies over less
precise ones
• Useful for getting the most accurate information about the population as a whole
• Not easily done in SPSS
34
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Reporting Statistical Models
• APA recommends exact p-values for all reported results; best to include an effect size, too– Effect “x” was not statistically significant in condition y, p
= .24, d = .21
• Report a mean and the upper and lower boundaries of the confidence interval as M = 30, 95% CI [20,40]– If all confidence intervals you are reporting are 95%, it’s
acceptable to say so and then later say something like:In this condition, effect x increased, M = 30 [20,40].
35
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Reliability Getting the same result when a measurement device is applied to the same quantity repeatedly.
Validity The extent to which a measurement tool (test, device) measures what it purports
to measure.
Control Behavior can be influenced by many factors, some known and others unknown to the researcher. Control refers to the systematic methods employed by a researcher to reduce threats to the validity of the study posed by extraneous influences on the behavior of both the participants and the observer.
Importance Does the research question we are trying to answer warrant the expenditure of resources (i.e., time, money,
effort) that will be required to complete the study).
Essential Elements of Research: Reliability, Validity, Control and Importance
36
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Types of Reliability
Test-retest ReliabilityConsistency of measurement over time
Internal Consistency Inter-item correlation
Interrater Reliability Level of agreement between independent observers of behavior(s). Assessed via correlation or the procedure at right.
AgreementAgreement + Disagreement
x 100
37
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Evaluating Measures: Effective Range
Effective Range: Scales sensitive enough to detect differences among one group of subjects may be insensitive to detect differences among another.
Scale Attenuation (or range restriction). A problem associated with scales not ranging high enough, low enough, or both.
Leads to “ceiling” effects and “floor” effects that distort data by not measuring the full range of a variable.
38
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Types of Validity
Face validity. The (non-empirical) degree to which a test appears to be a sensible measure.
Content validity. The extent to which a test adequately samples the domain of information, knowledge, or skill that it purports to measure.
Criterion validity. Now (concurrent) and Later (predictive). Involves determining the relationship (correlation) between the predictor (IV) and the criterion (DV).
Construct validity. The degree to which the theory or theories behind the research study provide(s) the best explanation for the results observed.
39
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Internal ValidityExtent to which causal/independent variable(s) and no other extraneous factors caused the change being measured.
External Validity (generalizability)Degree to which the results and conclusions of your study would hold for other persons, in other places, and at other times.
Internal vs. External Validity
40
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Selection
History
Maturation
Repeated Testing
Instrumentation
Regression to the mean
Subject mortality
Selection-interactions
Experimenter bias
Threats to Internal Validity:Factors that reduce our ability to draw valid conclusions
41
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
The role of ControlBehavior is influenced by many factors termed—confounding variables—that tend to distort the results of a study, thereby making it impossible for the researcher to draw meaningful conclusions. Some of these may be unknown to the researcher.
Control refers to the systematic methods (e.g., research designs) employed to reduce threats to the validity of the study posed by extraneous influences on both the participants and the observer (researcher).
Reducing Threats to Internal Validity
42
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Group/Selection threatOccurs when nonrandom procedures are used to assign subjects to conditions or when random assignment fails to balance out differences among subjects across the different conditions of the experiment.
Example:A researcher is interested in determining the factors most likely to elicit aggressive behavior in male college students. He exposes subjects in the experimental group to stimuli thought to provoke aggression and subjects in the control group to stimuli thought to reduce aggression and then measures aggressive behaviors of the students. How would the selection threat operate in this instance?
43
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
History threat
Events that happen to participants during the research which affect results but are not linked to the independent variable.
Example:
The reported effects of a program designed to improve medical residents’ prescription writing practices by the medical school may have been confounded by a self-directed continuing education series on medication errors provided to the residents by a pharmaceutical firm's medical education liaison.
44
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Maturation threat
Can operate when naturally occurring biological or psychological changes occur within subjects and these changes may account in part or in total for effects discerned in the study.
Example:A reported decrease in emergency room visits in a long-term study of pediatric patients with asthma may be due to subjects outgrowing childhood asthma rather than to any treatment regimen introduced to treat the asthma.
45
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Repeated testing threatMay occur when changes in test scores occur not because of the intervention but rather because of repeated testing. This is of particular concern when researchers administer identical pretests and posttests.
Example:
A reported improvement in medical resident prescribing behaviors and order-writing practices in the study previously described may have been due to repeated administration of the same short quiz. That is, the residents simply learned to provide the right answers rather than truly achieving improved prescribing habits.
46
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Instrumentation threat
When study results are due to changes in instrument calibration or observer changes rather than to a true treatment effect, the instrumentation threat is in operation.
Example:
In Kalsher’s Experimental Methods and Statistics course, he evaluates students progress in understanding principles of research design at week 3 of the semester. A graduate T.A. evaluates the students at the conclusion of the course. If the evaluators are dissimilar enough in their approach, perhaps because of lack of training, this difference may contribute to measurement error in trying to determine how much learning occurred over the semester.
47
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Statistical Regression threatThe regression threat can occur when subjects have been selected on the basis of extreme scores, because extreme (low and high) scores in a distribution tend to move closer to the mean (i.e., regress) in repeated testing.
Example:
if a group of subjects is recruited on the basis of extremely high stress scores and an educational intervention is then implemented, any improvement seen could be due partly, if not entirely, to regression to the mean rather than to the coping techniques presented in the educational program.
48
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Experimental Mortality threatExperimental mortality—also known as attrition, withdrawals, or dropouts—is problematic when there is a differential loss of subjects from comparison groups subsequent to randomization, resulting in unequal groups at the end of a study.
Example:
Suppose a researcher conducts a study to compare the effects of a corticosteroid nasal spray with a saline nasal spray in alleviating symptoms of allergic rhinitis (irritation and inflammation of the nasal passages). If subjects with the most severe symptoms preferentially drop out of the active treatment group, the treatment may appear more effective than it really is.
49
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Selection Interaction threatsA family of threats to internal validity produced when a selection threat combines with one or more of the other threats to internal validity. When a selection threat is already present, other threats can affect some experimental groups, but not others.
Example:If one group is dominated by members of one fraternity (selection threat), and that fraternity has a party the night before the experiment (history threat), the results may be altered for that group.
50
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
People, Places, and Times
Demand Characteristics
Hawthorne Effects
Order Effects (or carryover effects)
Threats to External Validity:Ways you might be wrong in making generalizations
51
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Example: You learn that the grant you submitted to assess average drinking rates among college students in the U.S. has been funded. In late November, you post an announcement about the study on campus to get subjects for the study. 100 students sign up for the study. Of these, 78 are members of campus fraternities; the other 22 are members of the school’s football team.
People threat:Are the results due to the unusual
type of people in the study?
52
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Example: Suppose that you conduct an “educational” study in a college town with lots of high-achieving educationally-oriented kids.
Places threat:Did the study work because of the unusual place you did the study in?
53
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Example: Suppose that you conducted a smoking cessation study the week after the U.S. Surgeon General issued the well publicized results of the latest smoking and cancer studies. In this instance, you might get different results than if you had conducted the study the week before.
Time threat:Was the study conducted at a peculiar time?
54
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Demand CharacteristicsParticipants are often provided with cues to the anticipated results of a study.
Example:
When asked a series of questions about depression, participants may become wise to the hypothesis that certain treatments may work better in treating mental illness than others. When participants become wise to anticipated results (termed a placebo effect), they may begin to exhibit performance that they believe is expected of them.
Making sure that subjects are not aware of anticipated outcomes (termed a blind study) reduces the possibility of this threat.
55
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Hawthorne Effects
Similar to a placebo, research has found that the mere presence of others watching a person’s performance causes a change in their performance. If this change is significant, can we be reasonably sure that it will also occur when no one is watching?
Addressing this issue can be tricky but employing a control group to measure the Hawthorne effect of those not receiving any treatment can be very helpful. In this sense, the control group is also being observed and will exhibit similar changes in their behavior as the experimental group therefore negating the Hawthorne effect.
56
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Order Effects (carryover effects)
Order effects refer to the order in which treatment is administered and can be a major threat to external validity if multiple treatments are used.
Example: If subjects are given medication for two months, therapy for another two months, and no treatment for another two months, it would be possible, and even likely, that the level of depression would be least after the final no treatment phase. Does this mean that no treatment is better than the other two treatments? It likely means that the benefits of the first two treatments have carried over to the last phase, artificially elevating the no treatment success rates.
57
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher 58
The Role of Experimental Design
In most social and behavioral research studies, we attempt to obtain at least one score from each participant (usually more!). Any obtained score is comprised of a number of components:
1. A ‘true score’ for the thing we hope we are measuring.
2. A ‘score for other things’ that we measure inadvertently.
3. Systematic (non-random) bias (usually ok as long as it affects all participants equally).
4. Random (non-systematic) error (which should cancel out over large numbers of observations).
We want our obtained score to consist of as much ‘true score’, and as little of the other factors, as possible.
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Research Study Control
Control removes sources of error in inferences– Reduces the chance of wrong conclusions– Increases the power of statistics to find
relationships in the presence of random error (“noise”)
Types of Control– Direct Manipulation– Randomization– Statistical Control
59
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Types of Control: Direct Manipulation
Sources of error held constant by research design or sampling decisions– Example: a researcher investigating the effects of
seeing justified violence in video games on children knows that young children cannot interpret the motives of characters accurately. She decides to limit her study to older children only, to eliminate random responses or unresponsiveness of younger children.
60
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Types of Control: Randomization
Unknown sources of error are equalized across all research conditions by randomly assigning subjects or by randomly choosing experimental materials.– Example: Many different factors are known to affect
the amount of use of Internet social networking sites. A researcher wants to test two different site designs. He randomly assigns subjects to work with each of the two designs. This equalizes the amount of confounding error from unknown factors in both groups.
61
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Known confounding variables are measured, and mathematical procedures are used to remove their effect.– Example: A political communication researcher
interested in studying emotional appeals versus rational appeals in political commercials suspects that the effects vary with the age of the viewer. She measures age, and uses it as an independent predictor (with multivariate statistics) to isolate, describe, and remove its effect.
62
Types of Control: Statistical Control
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Contrasting Methods of Control
Type of Control
Strength Weakness
Direct Manipulation
• Removes effect completely • Must know source of effect• Reduces generalizability
Randomization • Don’t have to know source of effect• Equalizes effect so there is no systematic confound
• Reduces statistical power by adding to unsystematic error variance
Statistical control
• Estimates effect of confounding variables• Expands theoretical model
• Must know source of effect• Requires more complex statistics
63
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher 64
Basic Types of Research
• Observational Methods
• Quasi-Experimental Designs
• True Experimental Designs
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Observational Methods
65
No direct manipulation of variables by the researcher. Behavior is merely recorded--but systematically and objectively so that the observations are potentially replicable.
Advantages• Reveals how people normally behave.• Experimentation without prior careful observation can lead to a
distorted or incomplete picture.
Disadvantages• Generally more time-consuming.• Doesn’t allow identification of cause and effect.
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Quasi-Experimental Design
66
In a quasi-experimental study, the experimenter does not have complete control over manipulation of the independent variable or how participants are assigned to the different conditions of the study.
Advantages• Natural setting• Higher face validity (from practitioner viewpoint)
Disadvantages• Not possible to isolate cause and effect as conclusively as with a
“true” experiment.
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher 67
Types of Quasi-Experimental Designs
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
One Group Post-Test Design
68
Treatment Measurement
Change in participants’ behavior may or may not be due to the intervention.
Prone to time effects, and lacks a baseline against which to measure the strength of the intervention.
Time
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
One Group Pre-test Post-test Design
69
Measurement Treatment Measurement
Comparison of pre- and post-intervention scores allows assessment of the magnitude of the treatment’s effects.
Prone to time effects, and it is not possible to determine whether performance would have changed without the intervention.
Time
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Interrupted Time-Series Design
70
Measurement
Measurement
Measurement
Treatment
Measurement
Measurement
Measurement
Time
Don’t have full control over manipulations of the IV. No way of ruling out other factors. Potential changes in measurement.
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Static Group Comparison Design
71
Treatment (experimental group)
Measurement
MeasurementNo Treatment
Group A:
Group B:(control group)
Participants are not assigned to the conditions randomly.
Observed differences may be due to other factors. Strength of conclusions depends on the extent to which we can identify and eliminate alternative explanations.
Time
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher 72
Experimental Research:
Between-Groups and Within-Groups Designs
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Between-Groups Designs
Separate groups of participant are used for each condition of the experiment.
Within-Groups (Repeated Measures) Designs Each participant is exposed to each condition of the experiment (requires less participants than between groups design).
73
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Between-Groups Designs
74
Advantages• Simplicity• Less chance of practice and fatigue effects• Useful when it is not possible for an individual to
participate in all of the experimental conditions
Disadvantages• Can be expensive in terms of time, effort, and number of
participants• Less sensitive to experimental manipulations
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Examples of Between-Groups Designs
75
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Post-test Only / Control Group Design
76
Treatment (experimental group)
Measurement
MeasurementNo Treatment
Group A:
Group B:(control group)
Randomallocation:
If randomization fails to produce equivalence, there is no way of knowing that it has failed. Experimenter cannot be certain that the two groups were comparable before the treatment.
Time
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Pre-test / Post-test Control Group Design
77
Treatment Measurement
No Treatment
Group A:
Group B:
Random
allocation:
Measurement
Measurement
Measurement
Pre-testing allows experimenter to determine equivalence of the groups prior to the intervention. However, pre-testing may affect participants’ subsequent performance.
Time
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Solomon Four-Group Design
78
Treatment Measurement
No Treatment
Group A:
Group B:
Ran
dom
allo
catio
n:
Measurement
Measurement
Measurement
Measurement
Measurement
Treatment
No Treatment
Group C:
Group D:
Time
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Within-Groups Designs: Repeated Measures
79
• Economy
• Sensitivity
Advantages
Disadvantages
• Carry-over effects from one condition to another
• The need for conditions to be reversible
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Repeated-Measures Design
80
Treatment Measurement
No Treatment
Random Allocation
Measurement
Measurement
Measurement
No Treatment
Treatment
Potential for carryover effects can be avoided by randomizing the order of presentation of the different conditions or counterbalancing the order in which participants experience them.
Time
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Latin Squares Design
81
One group of participants
Another group of participants
Yet another group of participants
A B C
B C A
C A B
order of conditions or trials:
Three Conditions or Trials
Order of presentation of conditions in a within-subjects design can be counterbalanced so that each possible order of conditions occurs just once. Problem not completely eliminated because A precedes B twice, but B precedes A only once. Same with C and A.
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Balanced Latin Squares Design
82
One group of participants
Another group of participants
Yet another group of participants
A B C D
B D A C
D C B A
order of conditions or trials:
And yet another group of participants C A D B
Four Conditions or Trials
Note: This approach works only for experiments with an even number of conditions. For additional help with more complex multi-factorial designs, see: http://www.jic.bbsrc.ac.uk
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Factorial Designs
83
• include multiple independent variables
• allow for analysis of interactions between variables
• facilitate increased generalizability
MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher
Important Concepts
84
Alternative hypothesis Dispersion Null hypothesis Score-level variable Standard Deviation
Between-groups design Effect Size Observational study Skew Standard Error
Categorical variable Experimental research One-tailed test Standard Deviation Systematic variation
Central tendency Face validity Ordinal variable Standard Error Two-tailed test
Confidence intervals Frequency distribution Outcome variable Systematic variation Type I error
Confounding variable Independent variable Platykurtic Two-tailed test Type II error
Construct validity Kurtosis Power Type I error Unsystematic variation
Content validity Leptokurtic Practice effects Type II error Validity
Continuous variable Level of Measurement Predictor variable Unsystematic variation Variance
Correlational research Mean Quasi-exp. research Validity Within-groups design
Counterbalancing Measurement error Randomization Variance z-scores
Criterion validity Median Range Within-groups design
Degrees of Freedom Mode Reliability z-scores
Dependent variable Nominal variable Repeated measures Score-level variable
Discrete variable Normal Distribution Sampling distribution Skew