INTRO 2 IRT Tim Croudace. 2 Descriptions of IRT “IRT refers to a set of mathematical models that...
-
date post
19-Dec-2015 -
Category
Documents
-
view
216 -
download
2
Transcript of INTRO 2 IRT Tim Croudace. 2 Descriptions of IRT “IRT refers to a set of mathematical models that...
INTRO 2 IRT
Tim Croudace
2
Descriptions of IRT• “IRT refers to a set of
mathematical models that describe, in probabilistic terms, the relationship between a person’s response to a survey question/test item and his or her level of the ‘latent variable’ being measured by the scale”
• Fayers and Hays p55– Assessing Quality of Life in
Clinical Trials. Oxford Univ Press: – Chapter on Applying IRT for
evaluating questionnaire item and scale properties.
• This latent variable is usually a hypothetical construct [trait/domain or ability] which is postulated to exist but cannot be measured by a single observable variable/item.
• Instead it is indirectly measured by using multiple items or questions in a multi-item test/scale.
3
The data:0000100000010010100110100011101101001100010101101101111001111111
n4776312
1507
32114
2319413
37812
1694531
Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!
logit {πhi} = αh 0 + αh 1zi
αh0 α10
αh0 α40
αh1 α21
4
Simple sum scores (n=1729 new individual values)
0 0 0 0 [n] Total score 0 0 0 0 477 0 477 zeros added to data set (new column) 1 0 0 0 63 1 0 0 0 1 12 1 0 0 1 0 150 1 1 0 0 1 7 2 1 0 1 0 32 2 0 0 1 1 11 2 1 0 1 1 4 3 0 1 0 0 231 1 1 1 0 0 94 2 0 1 0 1 13 2 0 1 1 0 378 2 1 1 0 1 12 3 1 1 1 0 169 3 0 1 1 1 45 3 1 1 1 1 31 4
5
Binary Factor / Latent Trait Analysis Results: logit-probit model
F
U1 U2 U3 Up. . .
Warming up to this sort of thing … soon ….
2 items with similar thresholds and similar slopes3 items with different thresholds but similar slopes
6
The key concept … latent factor models for constructs underpinning multiple
binary (0/1) responses• … based on innovations in educational testing
and psychometric statistics > 50 years old• Same models used in educational testing with
correct incorrect answers can be applied to symptom present / absent data (both binary)
• Extensions to ordinal outcomes (Likert scales)• Flexibility in parametric form available• Semi- and non-parametric approaches too…
7
Binary IRT : The A B C D of it
8
Linear vs non-linear regression of response probability on latent variable
x-axis score on latent construct being measured
y-axis
prob
of
response
(“Yes”)
on a
simple
binary
(Yes/No)
scale
item
Adapted without permissionfrom aslide by Prof H Goldstein
9
Ordinal IRT : The A B C D of GRM
10
IRT models• Simplest case of a latent trait analysis…– Manifest variables are binary: only 2 distinctions are made
• these take 0/1 values– Yes / No– Right / Wrong– Symptom present / absent
• Agree / disagree distinctions for attitudes more likely to be ordinal [>2 response categories] .. see next lecture IRT 2 on Friday
• For scoring of individuals – (not parameter estimation for items)
• it is frequently assumed that the UNOBSERVED (latent) variable < the latent factor / trait> • is not only continuous but normally distributed
– [or the prior dist’n is normal but the posterior dist’n may not be]
11
IRT for binary data The most commonly used model was developed by
Lord-Birnbaum model (Lord, 1952; Birnbaum, )
2-parameter logistic [a.k.a. the logit-probit model; Bartholomew (1987)]
• The model is essentially a non-linear single factor model– When applied to binary data, the traditional linear factor model is only an
approximation to the appropriate item response model• sometimes satisfactory, but sometimes very poor (we can guess when)
• Some accounts of Item Response Theory make it sound like a revolutionary & very modern development
• this is not true!– It should not replace or displace classical concepts, and has suffered from
being presented and taught as disconnected from these– A unified treatment can be given that builds one from the other (McDonald,
1999) but this would be a one term course on its own
12
What IRT does IRT models provide a clear statement [picture!] of the performance of each item in the scale/test
and
how the scale/test functions, overall,for measuring the construct of interest in the study population
The objective is to model each item by estimating the properties describing item performance characteristics
hence Item Characteristic Curve or Symptom Response Function.
13
Very bland (but simple) example
• Lombard and Doering (1947) data• Questions on cancer knowledge with four addressing
the source of the information• Fitting a latent variable model might be proposed as
a way of constructing a measure of how well informed an individual is about cancer
• A second stage might relate knowledge about cancer to knowledge about other diseases or general knowlege
14
Very bland (but simple) example
• Lombard and Doering (1947) data• Questions on cancer knowledge with four
addressing the source of the information– radio– newspapers– (solid) reading (books?)– lectures
• 2 to the power 4 i.e. 16 possible response patterns from 0000 to 1111
15
Data
• Lombard and Doering (1947) data
• 2 to the power 4 – i.e. 16 possible response
patterns (all occur)– with more items this is neither
likely nor necessary
– frequency shown for• 0000 to 1111• frequency is the number with
each item response pattern
0000100000010010100110100011101101001100010101101101111001111111
n4776312
1507
32114
2319413
37812
1694531
16
The data:0000100000010010100110100011101101001100010101101101111001111111
n4776312
1507
32114
2319413
37812
1694531
Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!
logit {πhi} = αh 0 + αh 1zi
αh0 α10
αh0 α40
αh1 α21
17
Basic objectives of modelling• When multiple items are applied in a test / survey can use
latent variable modelling to– explore inter-relationships among observed responses– determine whether the inter-relationships can be explained by a small
number of factors
– THEN , to assign a SCORE to each individual each on the basis of their responses
– Basically to rank order (arrange) or quantify (score) survey participants, test takers, individuals who have been studied» CAN BE THOUGHT OF AS ADDING A NEW SCORE TO YOUR DATASET FOR
EACH INDIVIDUAL• this analysis will also help you to understand the properties of each
item, as a measure of the target construct (what properties?)» GRAPHICAL REPRESENTATION IS BEST
18
Item Properties that we are interested in are captured graphically by so called Item Characteristics
Curves (ICCs)
19
Item/Symptom & Test/Scale INFORMATION– is useful and necessary to examine score precision (the
accuracy of estimated scores)– we are interested in this for different individuals
(individuals with different score values) – by inspecting the amount of information about each
score level, across the score range (range of estimated scores) we are identifying variations in measurement precision (reliable of individual’s estimated scores)
– this enables us to make statements about the effective measurement range of an instrument in an population
20
e.g. Item Characteristics Curves
21
Item information functions- add them together to get TIF
beware y axis scaling : not all the same
22
Test Information Function
23
Item information functions- shown alongside their ICCs
beware y axis scaling : not all the same
1111
Item Characteristics CurvesItem Characteristics Curves
0.14
0.40
3.0
0.14
24
1 / Sqrt [Information] = s.e.mInfo Sqrt(Info) 1/(sqrt(Info)1 1.0 1.02 1.4 0.73 1.7 0.64 2.0 0.55 2.2 0.46 2.4 0.47 2.6 0.48 2.8 0.49 3.0 0.310 3.2 0.311 3.3 0.312 3.5 0.3
Standard error of measuremenr is not constant (U-shaped, not symmetrical)
25
Approximate reliability
• Reliability= 1 – 1/[Info]
= {1 – 1 / [1 / (s.e.m ^2) }
s.e.m. = standard error of measurement
26
Back to the Data
• Lombard and Doering (1947) data
• 2 to the power 4 – i.e. 16 possible response
patterns (all occur)– with more items this is neither
likely nor necessary
– frequency shown for• 0000 to 1111• frequency is the number with
each item response pattern
0000100000010010100110100011101101001100010101101101111001111111
n4776312
1507
32114
2319413
37812
1694531
What would be the easiest thing to do with these numbers; to score the patterns..?
27
Answer ..0000100000010010100110100011101101001100010101101101111001111111
What would be the easiest thing to do with these numbers; to score the patterns..?
• Simply add them up
28
Simple sum scores (n=1729 new individual values)
0 0 0 0 [n] Total score 0 0 0 0 477 0 477 zeros added to data set (new column) 1 0 0 0 63 1 0 0 0 1 12 1 0 0 1 0 150 1 1 0 0 1 7 2 1 0 1 0 32 2 0 0 1 1 11 2 1 0 1 1 4 3 0 1 0 0 231 1 1 1 0 0 94 2 0 1 0 1 13 2 0 1 1 0 378 2 1 1 0 1 12 3 1 1 1 0 169 3 0 1 1 1 45 3 1 1 1 1 31 4
29
Weighted [by discriminating power] scores 0 0 0 0 [n] Total Factor Component [weighted by alpha h 1]
score score score 0 0 0 0 477 0 -0.98 0 = 0 1 0 0 0 63 1 -0.68 0.720.72 0.720.72 0 0 0 1 12 1 -0.67 0.770.77 0.770.77 0 0 1 0 150 1 -0.46 1.341.34 1.341.34 1 0 0 1 7 2 -0.41 0.720.72++ 0.770.77 1.481.48 1 0 1 0 32 2 -0.23 0.720.72 +1.341.34 2.062.06 0 0 1 1 11 2 -0.22 1.341.34++ 0.770.77 2.102.10 1 0 1 1 4 3 0.0 0.720.72++ 1.341.34++ 0.770.77 2.822.82 0 1 0 0 231 1 0.16 3.403.40 3.403.40 1 1 0 0 94 2 0.42 0.720.72++3.403.40 4.124.12 0 1 0 1 13 2 0.43 3.403.40++ 0.770.77 4.164.16 0 1 1 0 378 2 0.66 3.403.40++ 1.341.34 4.744.74 1 1 0 1 12 3 0.72 0.720.72++ 3.403.40++ 0.770.77 4.884.88 1 1 1 0 169 3 0.99 0.720.72++ 3.403.40++1.341.34 5.46 0 1 1 1 45 3 1.02 3.403.40++1.341.34++ 0.770.775.505.50 1 1 1 1 31 4 1.41 0.720.72++3.403.40++1.341.34++0.770.77 6.226.22
0.720.723.403.401.341.340.770.77
3737
Mplus version 4.1Mplus version 4.1 ML Estimate S.E. ML Estimate S.E.
Z by Q1 alpha h 1 0.721 0.093Z by Q1 alpha h 1 0.721 0.093Z by Q2 alpha h 2 3.358 1.035Z by Q2 alpha h 2 3.358 1.035Z by Q3 alpha h 3 1.344 0.167Z by Q3 alpha h 3 1.344 0.167Z by Q4 alpha h 4 0.769 0.145Z by Q4 alpha h 4 0.769 0.145
Variances Z 1Variances Z 1
Compare with Bartholomew (1987) p160Compare with Bartholomew (1987) p1600.72 (0.09) 0.72 (0.09) 3.40 (1.14)3.40 (1.14)1.34 (0.17)1.34 (0.17)0.77 (0.15)0.77 (0.15)
30
The data:0000100000010010100110100011101101001100010101101101111001111111
n4776312
1507
32114
2319413
37812
1694531
Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!
logit {πhi} = αh 0 + αh 1zi
αh0 α10
αh0 α40
αh1 α21
31
Weighted [by discriminating power] scores 0 0 0 0 [n] Total Factor Component [weighted by alpha h 1]
score score score 0 0 0 0 477 0 -0.98 0 = 0 1 0 0 0 63 1 -0.68 0.720.72 0.720.72 0 0 0 1 12 1 -0.67 0.770.77 0.770.77 0 0 1 0 150 1 -0.46 1.341.34 1.341.34 1 0 0 1 7 2 -0.41 0.720.72++ 0.770.77 1.481.48 1 0 1 0 32 2 -0.23 0.720.72 +1.341.34 2.062.06 0 0 1 1 11 2 -0.22 1.341.34++ 0.770.77 2.102.10 1 0 1 1 4 3 0.0 0.720.72++ 1.341.34++ 0.770.77 2.822.82 0 1 0 0 231 1 0.16 3.403.40 3.403.40 1 1 0 0 94 2 0.42 0.720.72++3.403.40 4.124.12 0 1 0 1 13 2 0.43 3.403.40++ 0.770.77 4.164.16 0 1 1 0 378 2 0.66 3.403.40++ 1.341.34 4.744.74 1 1 0 1 12 3 0.72 0.720.72++ 3.403.40++ 0.770.77 4.884.88 1 1 1 0 169 3 0.99 0.720.72++ 3.403.40++1.341.34 5.46 0 1 1 1 45 3 1.02 3.403.40++1.341.34++ 0.770.775.505.50 1 1 1 1 31 4 1.41 0.720.72++3.403.40++1.341.34++0.770.77 6.226.22
0.720.723.403.401.341.340.770.77
3737
Mplus version 4.1Mplus version 4.1 ML Estimate S.E. ML Estimate S.E.
Z by Q1 alpha h 1 0.721 0.093Z by Q1 alpha h 1 0.721 0.093Z by Q2 alpha h 2 3.358 1.035Z by Q2 alpha h 2 3.358 1.035Z by Q3 alpha h 3 1.344 0.167Z by Q3 alpha h 3 1.344 0.167Z by Q4 alpha h 4 0.769 0.145Z by Q4 alpha h 4 0.769 0.145
Variances Z 1Variances Z 1
Compare with Bartholomew (1987) p160Compare with Bartholomew (1987) p1600.72 (0.09) 0.72 (0.09) 3.40 (1.14)3.40 (1.14)1.34 (0.17)1.34 (0.17)0.77 (0.15)0.77 (0.15)
32
Something a little more subtle
• Simple sum scores assumes all item responses equally useful at defining the construct– may not be the case
• If items are differentially important– different discriminating power with respect to what we
are measuring, we might want to take that into accounf• How? Weighted sum scores [Component scores]
– weighted by what?» weighted by the estimates (factor loading type parameter) from
a latent variable model» [latent trait model with a single latent factor]
33
Weightedscores
3737
Mplus version 4.1Mplus version 4.1 ML Estimate S.E. ML Estimate S.E.
Z by Q1 alpha h 1 0.721 0.093Z by Q1 alpha h 1 0.721 0.093Z by Q2 alpha h 2 3.358 1.035Z by Q2 alpha h 2 3.358 1.035Z by Q3 alpha h 3 1.344 0.167Z by Q3 alpha h 3 1.344 0.167Z by Q4 alpha h 4 0.769 0.145Z by Q4 alpha h 4 0.769 0.145
Variances Z 1Variances Z 1
Compare with Bartholomew (1987) p160Compare with Bartholomew (1987) p1600.72 (0.09) 0.72 (0.09) 3.40 (1.14)3.40 (1.14)1.34 (0.17)1.34 (0.17)0.77 (0.15)0.77 (0.15)
2626
CancerKnowledge zi
The data:0000100000010010100110100011101101001100010101101101111001111111
n4776312150732114
2319413378121694531
Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!
logit {πhi} = αh 0 + αh 1zi
αh0 α10
αh0 α40
αh1 α21
WeightsWeightsalpha h 1 alpha h 1 parametersparameters
Q1Q1 0.720.72Q2Q2 3.403.40Q3Q3 1.341.34Q4Q4 0.770.77These numbers These numbers related to the related to the slopes of the S’sslopes of the S’s
2020
Item information functionsItem information functions-- shown alongside their shown alongside their ICCsICCs
beware y axis scaling : not all the same
1111
Item Characteristics CurvesItem Characteristics Curves
0.14
0.40
3.0
0.14
34
Estimated component scores (weighted values)
0 0 0 0 [n] Total Factor Component [weighted by alpha h 1]
score score score 0 0 0 0 477 0 -0.98 0 = 0 1 0 0 0 63 1 -0.68 0.720.72 0.720.72 0 0 0 1 12 1 -0.67 0.770.77 0.770.77 0 0 1 0 150 1 -0.46 1.341.34 1.341.34 1 0 0 1 7 2 -0.41 0.720.72++ 0.770.77 1.481.48 1 0 1 0 32 2 -0.23 0.720.72 +1.341.34 2.062.06 0 0 1 1 11 2 -0.22 1.341.34++ 0.770.77 2.102.10 1 0 1 1 4 3 0.0 0.720.72++ 1.341.34++ 0.770.77 2.822.82 0 1 0 0 231 1 0.16 3.403.40 3.403.40 1 1 0 0 94 2 0.42 0.720.72++3.403.40 4.124.12 0 1 0 1 13 2 0.43 3.403.40++ 0.770.77 4.164.16 0 1 1 0 378 2 0.66 3.403.40++ 1.341.34 4.744.74 1 1 0 1 12 3 0.72 0.720.72++ 3.403.40++ 0.770.77 4.884.88 1 1 1 0 169 3 0.99 0.720.72++ 3.403.40++1.341.34 5.46 0 1 1 1 45 3 1.02 3.403.40++1.341.34++ 0.770.775.505.50 1 1 1 1 31 4 1.41 0.720.72++3.403.40++1.341.34++0.770.77 6.226.22
??????????0.720.723.403.401.341.340.770.77
35
But the bees knees are..
• The estimated factor scores from the model• Not just some simple sum or unweighted or
weighted items• Takes into account the proposed score distribution
(gaussian normal) and the estimated model parameters (but not the fact that they are estimates rather than known values) and more besides (when missing data are present)
… the estimated factor scores
36
A graphical and interactiveintroduction to IRT
• Play with the key features of IRT models
• www2.uni-jena.de/svw/metheval/irt/VisualIRT.pdf
37
a b (see) [2 parameter IRT model]
• VisualIRT (pdf)– Page
• VisualIRT (pdf)– Page
Individual’s score = new ruler valueAny hypothetical latent variable [factor/trait] continuum expressed in a z-score metric (gaussian normal (0,1)
Item propertiesslope = item discriminationlocation = item commonality [difficulty/prevalance/ severity]
38
IRT Resources• A visual guide to Item Response Theory
– I. Partchev• Introduction to RIT,
– R.Baker• http //ericae.net/irt/baker/toc.htm
• An introduction to modern measurement theory– B Reeve
• Chapter in Fayers and Machin QoL book– P Fayers
• ABC of Item Response Theory– H Goldstein
• Moustaki papers, and online slides (FA at 100)• LSE books (Bartholomew, Knott, Moustaki, Steele)
39
Item Response Theory Books
Applications of Item Response Theory to Practical Testing Problems Frederick M. Lord. 274 pages. 1980. Applying The Rasch Model Trevor G. Bond and Christine M. Fox 255 pages. 2001. Constructing Measures: An Item Response Modeling Approach Mark Wilson. 248 pages. 2005. The EM Algorithm and Related Statistical Models Michiko Watanabe and Kazunori Yamaguchi. 250 pages. 2004. Essays on Item Response Theory Edited by Anne Boomsma, Marijtje A.J. van Duijn, Tom A.A. Snijders. 438 pages. 2001. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach Edited by Paul De Boeck and Mark Wilson. 382 pages. 2004. Fundamentals of Item Response Theory Ronald K. Hambleton, H. Swaminathan, and H. Jane Rogers. 184 pages. 1991. Handbook of Modern Item Response Theory Edited by Wim J. van der Linden and Ronald K. Hambleton. 510 pages. 1997. Introduction to Nonparametric Item Response Theory Klaas Sijtsma and Ivo W. Molenaar. 168 pages. 2002. Item Response Theory Mathilda Du Toit. 906 pages. 2003. Item Response Theory for Psychologists Susan E. Embretson and Steven P. Reise. 376 pages. 2000. Item Response Theory: Parameter Estimation Techniques (Second Edition, Revised and Expanded w/CD) Frank Baker and Seock-Ho Kim. 495 pages. 2004. Item Response Theory: Principles and Applications Ronald K. Hambleton and Hariharan Swaminathan. 332 pages. 1984. Logit and Probit: Ordered and Multinomial Models Vani K. Borooah. 96 pages. 2002. Markov Chain Monte Carlo in Practice W.R. Gilks, Sylvia Richardson, and D.J. Spiegelhalter. 512 pages. 1995. Monte Carlo Statistical Methods Christian P. Robert and George Casella. 645 pages. 2004. Polytomous Item Response Theory Models Remo Ostini and Michael L. Nering. 120 pages. 2005. Rasch Models for Measurement David Andrich. 96 pages. 1988. Rasch Models: Foundations, Recent Developments, and Applications Edited by Gerhard H. Fischer and Ivo W. Molenaar. 436 pages. 1995. The Sage Handbook of Quantitative Methodology for the Social Sciences Edited by David Kaplan. 511 pages. 2004. Test Equating, Scaling, and Linking: Methods and Practices (Second Edition) Michael J. Kolen and Robert L. Brennan. 548 pages. 2004.