Rasch trees: A new method for detecting differential item functioning in the Rasch model
Differential Item Functioning
description
Transcript of Differential Item Functioning
Differential Item Functioning
Laura Gibbons, PhD
Thank you, Rich Jones, Frances Yang, and Paul Crane.
Thank you, NIA• Funding for this conference was made possible, in
part by Grant R13AG030995-01A1 from the National Institute on Aging. The views expressed in written conference materials or publications and by speakers and moderators do not necessarily reflect the official policies of the Department of Health and Human Services; nor does mention by trade names, commercial practices, or organizations imply endorsement by the U.S. Government.
Many (most?) of these slides were adapted or copied directly from his presentations.Check out his 3-day workshop:http://www.hebrewseniorlife.org/latent-variable-methods-workshop
Thank you, Rich
Outline
1. Why do we care about DIF?2. A few notes about Item response theory3. What is DIF?4. How do we find DIF?5. What do we do when we find DIF?6. Does DIF matter?
We want unbiased tests
• We want a test score to mean the same thing in all subgroups of people.
• Test bias has been recognized as an issue for at least a century.– Missing a question based on a reference to a
regatta may indicate race and/or SES, not intelligence.
• Test bias came to the forefront in the 60’s, particularly with respect to race. – Many similar assumptions of a uniform culture
turned out to be invalid.– Educational testing, intelligence testing, insurance
licensing, firefighting– There hasn’t been a big political struggle for lack
of bias in cognitive aging, measures of affect, but an important research concern none the less.
My favorite example of potential bias
• Does endorsing the item “I cry easily” mean the same thing in women as in men?
Cognitive Tests
• Cognitive test scores should represent cognition, not sex, race, test language, age, SES, etc.
• True differences between groups in cognition exist.
• However, the difference should not affect the relationship between a person’s cognitive test score and their true cognitive ability.
2. A few notes about Item Response Theory
Key Ideas of IRT• Persons have a certain ability or trait• Items have characteristics
– difficulty (how hard the item is)– discrimination (how well the item measures the ability)– (I won’t talk about guessing)
• Person ability, and item characteristics are estimated simultaneously and expressed on unified metric
• Interval-level measure of ability or trait. – This means that no matter what your ability level, a change
of one point in the score represents an equivalent amount of change in ability. (NOT true for MMSE and most cognitive tests.)
Some Things Rich (and others)Can Do with IRT
1. Refine measures2. Identify ‘biased’ test items3. Adaptive testing4. Handle missing data at the item level5. Equate measures
Latent Ability / Trait• Symbolized with qi (or hi)
• Assumed to be continuously, and often normally, distributed in the population
• The more of the trait a person has, the more likely they are to ...whatever...(endorse the symptom, get the answer right etc.)
• The latent trait is that unobservable, hypothetical construct presumed to be measured by the test (assumed to “cause” item responses)
Dimensionality• It matters whether or not the latent trait is
unidimensional.– Knowing a person’s level on single underlying latent trait is
sufficient to predict their likelihood of success on an item.– Item responses are dependent upon a person’s ability (and
item characteristics) only.– Secondary factors are trivial.
• There are methods that allow for departures from unidimensionality, but I won’t talk about them today.
Item Characteristic Curve• The fundamental conceptual unit of IRT
• Relates item responses to the ability presumed to cause them
• Represented with cumulative logistic or cumulative normal distributions
• Here we illustrate with dichotomous items, for simplicity
Item Response Function
P(yij=1|qi) = F[aj(qi-bj)]Example of an Item Characteristic Curve
Prob
abili
ty o
f Cor
rect
Res
pons
e
Latent Ability Distribution-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Example of an Item Characteristic Curve: High AbilityExample of an Item Characteristic Curve
Prob
abili
ty o
f Cor
rect
Res
pons
e
Latent Ability Distribution-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
A Person with High AbilityHas a High Probability ofResponding Correctly
Example of an Item Characteristic Curve: Low AbilityExample of an Item Characteristic Curve
Prob
abili
ty o
f Cor
rect
Res
pons
e
Latent Ability Distribution-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
A Person with Low AbilityHas a Low Probability ofResponding Correctly
Item Difficulty
Example of an Item Characteristic Curve
Prob
abili
ty o
f Cor
rect
Res
pons
e
Latent Ability Distribution-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Item Difficulty: The level ofability at which a person hasa 50% probability ofresponding correctly.
Item and Person Ability are on the Same Metric
LatentTraitDensity
0.000.100.200.300.400.500.600.700.800.901.00
Prob
abili
ty o
f a C
orre
ct R
espo
nse
-3 -2 -1 0 1 2 3Latent Trait Level
Example of Two ICCs that Differ in Difficulty
Prob
abili
ty o
f Cor
rect
Res
pons
e
Latent Ability Distribution-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Item Discrimination
Example of an Item Characteristic Curve
Prob
abili
ty o
f Cor
rect
Res
pons
e
Latent Ability Distribution-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Item Discrimination:How well the item separatespersons of high and low ability;Proportional to the slope of theICC at the point of inflection
The Steeper Curve Has Greater Discrimination
Example of Two ICCs that Differ in Discrimination
Prob
abili
ty o
f Cor
rect
Res
pons
e
Latent Ability Distribution-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
3. What is DIF?
Identify Biased Test ItemsDifferential Item Functioning (DIF)
• Differences in endorsing a given item may be due to – group differences in ability– item bias– both
• IRT can parse this out• Item Bias = Differential Item Function + Rationale• Most IRT users identify DIF when two groups do not
have the same ICC
• DIF: When a demographic characteristic
interferes with the relationship expected between a person’s ability level and responses to an item.
• This is a conditional definition; we have to control for ability level, or else we can’t differentiate between DIF and differential test impact.
LatentTraitDens ity
0.000.100.200.300.400.500.600.700.800.901.00
Pro
babi
lity
of a
Cor
rect
Res
pons
e
-3 -2 -1 0 1 2 3Latent Trait Level
Example of group heterogeneity but no DIF
Here the overall levels differ, and there is also Uniform DIF
LatentTraitDens ity
0.000.100.200.300.400.500.600.700.800.901.00
Pro
babi
lity
of a
Cor
rect
Res
pons
e
-3 -2 -1 0 1 2 3Latent Trait Level
Example of group heterogeneity and uniform DIF
Non-Uniform (and uniform) DIF
LatentTraitDens ity
0.000.100.200.300.400.500.600.700.800.901.00
Pro
babi
lity
of a
Cor
rect
Res
pons
e
-3 -2 -1 0 1 2 3Latent Trait Level
Example of group heterogeneity and non-uniform DIF
4. How do we find DIF?
Chi-square
• Educational testing still uses 2x2 tables and chi-squared tests.
• Pros: conceptually and computationally easy• Cons:
– Needs huge samples with adequate discordance.– Need to estimate ability and DIF in separate steps,
potentially introducing bias.– Assumes ability is unidimensional.
Logistic Regression
• Logistic regression, or ordinal logistic regression for ordinal items.
• Uses the logistic link for the ICC curve equation:
P(yij=1|qi) = F[aj(qi-bj)]
The 2 Parameter Logistic model
• Logit P(Y=1|a,b,θ)=Da(θ-b)– Models probability that a person correctly
responds to an item given the item parameters (a,b) and their person ability level θ
– b is the item difficulty• When θ=b, 50% probability of getting the item correct
– a is item discrimination• a determines slope around the point where θ=b
– D is a constant
1. P(Y=1| θ)=f(β1 θ)2. P(Y=1| θ, group)=f(β1 θ +β2*group)3. P(Y=1| θ, group)=f(β1 θ +β2*group+β3* θ *group)
– Uniform DIF: Compare models 1 and 2.– Non-Uniform DIF: Compare models 2 and 3.
Logistic Regression
• Pros: – Handles fairly small samples. – Quick and easy if you’ve got Stata and Parscale, or
R• Cons:
– Need to estimate ability and DIF in separate steps, potentially introducing bias.
– Assumes ability is unidimensional.– Need specific software.
Logistic Regression
Latent Variable Modeling
• Single and 2-group MIMIC* models.• “We” use Mplus for this.• Compare the loadings and intercepts of the
test items.
* Multiple Indicators Multiple Causes
Factor Analysis with Covariates
x
h
1 1y
4y
1
2y 2
3
4
2
3
4y 3
*
*
*
*
1
1 ,1
1 ,1
1
11 , 1
MIMIC Model Multiple Indicators, Multiple Cause
y = h + x +
assuming VAR(h) = 1, h=0
a = 1-2 , b =
x
is sufficient to describe uniform DIF
Multiple Group (MG) MIMIC
x
h
1 1y
4y
1
2y 2
3
4
2
3
4y 3
*
*
*
*
1
1 ,1
1 ,1
1
11 , 1
x
h
1 1y
4y
1
2y 2
3
4
2
3
4y 3
*
*
*
*
1
1 ,1
1 ,1
1
11 , 1
gro u p = 0 gro u p = 1
• Pros:– Simultaneous modeling of differences in ability and item-level
performance– Capable of handling multidimensional constructs– Can use continuous variables for Uniform DIF
• Cons:– Not precisely the IRT model– Modeling Non-Uniform DIF a challenge (Multiple Group
models required)– Need specialized software.
Latent Variable Modeling
5. What do we do when we find DIF?
• In educational settings, often items with DIF are discarded.
• Unattractive option for us– Tests are too short as it is.– Lose variation and precision.– DIF doesn’t mean that the item doesn’t measure
the underlying construct at all, just that it does so differently in different groups.
Discard the item?
Better to account for the DIF
• In logistic regression:• Constrain parameters for DIF-free items to be
identical across groups• Estimate parameters for items found with DIF
separately in appropriate groups
• In latent variable modeling, it’s all one big model.
If we account for DIF, is the test unbiased?
• We can only adjust for measured covariates.• Confounders such as education level may
mean different things for different groups.• We may lack power or the data may be too
sparse to account for all the DIF.• If most of the items on a test are biased, it’s
hard to get anywhere.
6. Does DIF matter?
DIF Impact
• We find DIF in a lot of cognitive tests.
• It’s important to assess the impact of items with DIF on the final score.
• Often DIF in individual items favors one group in some items and the other group in others, the net result being a score that has little bias.
Good for the field, bad for my job security
• So far, in my experience, cognitive scores accounting for DIF correlate very highly with the original IRT scores.
– Even for DIF with respect to test language.
Here at Friday Harbor
• How about depression scales?My workgroup will look.
• Alden’s calibrated scores?Fascinating missing data question.
Despite what I said about usually finding minimal impact,
DIF should be assessed as part of any test validation.