Post on 22-Jun-2020
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 1
/ 33
On the statistical meaning of the item parame-ters in IRT modelsCONBRATRI VI
Ernesto San Martín1Faculty of Mathematics, Pontificia Universidad Católica de Chile, Chile2The Economics School of Louvain, Université catholique de Louvain, Belgium3Laboratorio Interdisciplinario de Estadística Social, LIES, Pontificia Universidad Católica de Chile, Chile
Agosto 2018, Juiz de Fora, Brasil
What means to model aeducational phenomenon?
We observe the response patterns of 50 students on 7 items:
It1 It2 It3 It4 It5 It6 It7Student 1 1 0 0 1 1 0 1Student 2 0 1 1 0 1 1 0Student 3 1 1 1 1 0 1 0Student 4 0 0 1 0 0 0 1Student 5 0 0 0 1 1 1 1Student 6 1 0 0 1 1 0 1Student 7 0 1 1 0 0 1 0Student 8 1 1 0 0 1 1 0Student 9 0 0 1 1 0 0 0Student 10 1 0 0 1 1 1 1
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 2
/ 33
What means to model aeducational phenomenon?
We observe the response patterns of 50 students on 7 items:
It1 It2 It3 It4 It5 It6 It7Student 1 1 0 0 1 1 0 1Student 2 0 1 1 0 1 1 0Student 3 1 1 1 1 0 1 0Student 4 0 0 1 0 0 0 1Student 5 0 0 0 1 1 1 1Student 6 1 0 0 1 1 0 1Student 7 0 1 1 0 0 1 0Student 8 1 1 0 0 1 1 0Student 9 0 0 1 1 0 0 0Student 10 1 0 0 1 1 1 1
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 2
/ 33
What means to model aeducational phenomenon?
We intend to describe the behavior of items.
We intend to describe the behavior of students.
In particular, we intend to identify test fraud as answer copyingbetween two examinees.
We can face these questions using psychometric models.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 3
/ 33
What means to model aeducational phenomenon?
We intend to describe the behavior of items.
We intend to describe the behavior of students.
In particular, we intend to identify test fraud as answer copyingbetween two examinees.
We can face these questions using psychometric models.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 3
/ 33
What means to model aeducational phenomenon?
We intend to describe the behavior of items.
We intend to describe the behavior of students.
In particular, we intend to identify test fraud as answer copyingbetween two examinees.
We can face these questions using psychometric models.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 3
/ 33
Detecting answer copyingand IRT models
IRT models: student’s answer on an item depends on both anindividual characteristic (ability) and item characteristics(difficulty, discrimination, guessing).
3PL model: P(Ypi = 1) = ci + (1− ci )exp(αiθp − βi )
1+ exp(αiθp − βi ).
If αi = 1 then we obtain the 1PL-G model.If ci = 0 then we obtain the 2PL model.If αi = 1 and ci = 0 then we obtain the 1PL model.
There are other extensions: the 4PL Model . . .
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 4
/ 33
Detecting answer copyingand IRT models
IRT models: student’s answer on an item depends on both anindividual characteristic (ability) and item characteristics(difficulty, discrimination, guessing).
3PL model: P(Ypi = 1) = ci + (1− ci )exp(αiθp − βi )
1+ exp(αiθp − βi ).
If αi = 1 then we obtain the 1PL-G model.If ci = 0 then we obtain the 2PL model.If αi = 1 and ci = 0 then we obtain the 1PL model.
There are other extensions: the 4PL Model . . .
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 4
/ 33
Detecting answer copyingand IRT models
IRT models: student’s answer on an item depends on both anindividual characteristic (ability) and item characteristics(difficulty, discrimination, guessing).
3PL model: P(Ypi = 1) = ci + (1− ci )exp(αiθp − βi )
1+ exp(αiθp − βi ).
If αi = 1 then we obtain the 1PL-G model.If ci = 0 then we obtain the 2PL model.If αi = 1 and ci = 0 then we obtain the 1PL model.
There are other extensions: the 4PL Model . . .
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 4
/ 33
Detecting answer copyingand IRT models
IRT models: student’s answer on an item depends on both anindividual characteristic (ability) and item characteristics(difficulty, discrimination, guessing).
3PL model: P(Ypi = 1) = ci + (1− ci )exp(αiθp − βi )
1+ exp(αiθp − βi ).
If αi = 1 then we obtain the 1PL-G model.If ci = 0 then we obtain the 2PL model.If αi = 1 and ci = 0 then we obtain the 1PL model.
There are other extensions: the 4PL Model . . .
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 4
/ 33
−4 −2 0 2 4
0.0
0.2
0.4
0.6
0.8
1.0
Pro
babi
lidad
de
resp
uest
a co
rrec
ta
beta=0beta=1beta=−1
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 5
/ 33
●●
●●
●
●●
●●●●●●●●●●●●●●●●
●●●
●●
●●
●
●
●
●
●●
●
●●
●
●
●●●
●
10 20 30 40
0.0
0.2
0.4
0.6
0.8
1.0
DE=0.31; CIT=0.37
Puntaje Total
Val
or d
e C
CE
I
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 6
/ 33
Detecting answer copyingand IRT models
Zopluoglu (2016) analyzes the statistical behavior of answer-copying indexes underdifferent IRT models (dichotomous and polychotomous)
He remarks that copying indexes share the same rationale, but the computation ofthe probability of choosing the alternative k of the item j for the person p is doneusing either an IRT model or the CTT framework.
The objective of his contribution is to analyze the type I error behavior and thestatistical power of copying indexes.
Data: 40 items of mathematics applied to 67896 students.
Adjust 1PL, 2PL and 3PL using IRTPRO (which provides MML estimators for theparameters that characterize the items, although it also provides Bayesianestimates).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 7
/ 33
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
● ●●
●
●
●
●●
●
●
●●
●
●●
−4 −3 −2 −1 0 1
−4
−3
−2
−1
01
Con datos de Zopluoglu (2016)
Dificultades 1PL
Difi
culta
des
2PL/
3PL
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
oo
1PL vs 2PL (0.81)1PL vs 3PL (0.86)
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 8
/ 33
Detecting answer copyingand IRT models
Study 1: manipulate sample sizes, amount of copy, type of copy, levels of itemsdifficulties.
Item difficulties: easy and medium difficulties . . . “overall test difficulty wasmanipulated through the b parameters for the dichotomous IRT models” (p.595).
Remark: Given the high correlations between the estimators of the difficulties, it ispossible to keep this part of the design comparable with respect to the different IRTmodels . . .
¿or not? ¿why?
For easy test difficulty conditions, test difficulty was around .80 for typical simulated responsedata which was similar to the real dataset. For medium test difficulty conditions, the b parametersfor dichotomous IRT models [. . . ] were manipulated such that test difficulty was around .50for typical simulated response data. This was accomplished by adding a constant of 1.52 forthe 1PL, 1.56 for the 2PL, and 1.53 for the 3PL model to the b parameters used for the easyconditions (pp.595-596.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 9
/ 33
Detecting answer copyingand IRT models
Study 1: manipulate sample sizes, amount of copy, type of copy, levels of itemsdifficulties.
Item difficulties: easy and medium difficulties . . . “overall test difficulty wasmanipulated through the b parameters for the dichotomous IRT models” (p.595).
Remark: Given the high correlations between the estimators of the difficulties, it ispossible to keep this part of the design comparable with respect to the different IRTmodels . . . ¿or not? ¿why?
For easy test difficulty conditions, test difficulty was around .80 for typical simulated responsedata which was similar to the real dataset. For medium test difficulty conditions, the b parametersfor dichotomous IRT models [. . . ] were manipulated such that test difficulty was around .50for typical simulated response data. This was accomplished by adding a constant of 1.52 forthe 1PL, 1.56 for the 2PL, and 1.53 for the 3PL model to the b parameters used for the easyconditions (pp.595-596.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 9
/ 33
Detecting answer copyingand IRT models
Study 1: manipulate sample sizes, amount of copy, type of copy, levels of itemsdifficulties.
Item difficulties: easy and medium difficulties . . . “overall test difficulty wasmanipulated through the b parameters for the dichotomous IRT models” (p.595).
Remark: Given the high correlations between the estimators of the difficulties, it ispossible to keep this part of the design comparable with respect to the different IRTmodels . . . ¿or not? ¿why?
For easy test difficulty conditions, test difficulty was around .80 for typical simulated responsedata which was similar to the real dataset. For medium test difficulty conditions, the b parametersfor dichotomous IRT models [. . . ] were manipulated such that test difficulty was around .50for typical simulated response data. This was accomplished by adding a constant of 1.52 forthe 1PL, 1.56 for the 2PL, and 1.53 for the 3PL model to the b parameters used for the easyconditions (pp.595-596.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 9
/ 33
Detecting answer copyingand IRT models
Copying types:
Random copying: it is assumed that the student who copies, copies responsesfrom a student-source in a random manner, so that all items are equally likelyto be copied.
Difficulty weighted copying: the items are ordered from the easiest to the moredifficult. The probability that each item is copied is proportional to their rank.
Outcome of interest: the AUC is used as a measure ofclassification accuracy for how well an index separates theanswer-copying and honest pairs of students.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 10
/ 33
Detecting answer copyingand IRT models
Results:
Amount of copy is the most important factor that affect the classification’sperformance.
The difficulty level of items and the copy types has a negligible effect.
It was expected that the indexes would be reduced to some degree given thepresence of guessing, but it doesn’t.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 11
/ 33
Critical review?
Initial question: What means to model an educationalphenomenon?
It means to provide structural definitions of educational concepts
either by introducing explicit structural definitions,
or by introducing statistical models the parameters of which“represent some properties of the population under analysis”(Fisher, 1922).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 12
/ 33
Critical review?
Initial question: What means to model an educationalphenomenon?
It means to provide structural definitions of educational concepts
either by introducing explicit structural definitions,
or by introducing statistical models the parameters of which“represent some properties of the population under analysis”(Fisher, 1922).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 12
/ 33
Critical review?
Initial question: What means to model an educationalphenomenon?
It means to provide structural definitions of educational concepts
either by introducing explicit structural definitions,
or by introducing statistical models the parameters of which“represent some properties of the population under analysis”(Fisher, 1922).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 12
/ 33
More questions . . .
What do we mean by the expression difficulty of an item in anIRT model?
What do we mean by guessing parameter?
How can we structurally define copying? We are ready to believethat some “irregular” patterns can be discovered” by statisticaltechnique? How can we found something that we didn’t definepreviously?
More fundamentally, can we measure without having a theoryabout what we intend to measure?
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 13
/ 33
More questions . . .
What do we mean by the expression difficulty of an item in anIRT model?
What do we mean by guessing parameter?
How can we structurally define copying? We are ready to believethat some “irregular” patterns can be discovered” by statisticaltechnique? How can we found something that we didn’t definepreviously?
More fundamentally, can we measure without having a theoryabout what we intend to measure?
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 13
/ 33
More questions . . .
What do we mean by the expression difficulty of an item in anIRT model?
What do we mean by guessing parameter?
How can we structurally define copying? We are ready to believethat some “irregular” patterns can be discovered” by statisticaltechnique? How can we found something that we didn’t definepreviously?
More fundamentally, can we measure without having a theoryabout what we intend to measure?
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 13
/ 33
Statistical model
A statistical model fully describe the observations.
A parameter ω captures features of the observations only if it isidentified –namely, if the mapping ω 7−→ Pω is injective.
ω > Pω
Observations∨
∧
>
Therefore, only identified parameters can be statisticallyinterpreted.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 14
/ 33
Statistical model
A statistical model fully describe the observations.
A parameter ω captures features of the observations only if it isidentified –namely, if the mapping ω 7−→ Pω is injective.
ω > Pω
Observations∨
∧
>
Therefore, only identified parameters can be statisticallyinterpreted.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 14
/ 33
Statistical model
A statistical model fully describe the observations.
A parameter ω captures features of the observations only if it isidentified –namely, if the mapping ω 7−→ Pω is injective.
ω > Pω
Observations∨
∧
>
Therefore, only identified parameters can be statisticallyinterpreted.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 14
/ 33
Identifiability andthe Bayesian approach
Let Y | ψ, λ) ∼ N (ψ + λ, σ2Y ) be the statistical model(likelihood).
The parameters (ψ, λ) are not identified.
Prior distribution: ψ ∼ N (µψ, σ2ψ), λ ∼ N (µλ, σ
2λ), and ψ ⊥⊥ λ.
We can compute the posterior distribution of ψ | Y and thereforethe problem seems to be solved: we learn on a non-identifiedparameter.
However, the posterior distribution of ψ is a function of theposterior distribution of the identified parameter because
E [ψ | Y ] = ηψ,λ +σ2ψ
σ2ψ + σ2λE [ψ + λ | Y ].
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 15
/ 33
Identifiability andthe Bayesian approach
Let Y | ψ, λ) ∼ N (ψ + λ, σ2Y ) be the statistical model(likelihood).
The parameters (ψ, λ) are not identified.
Prior distribution: ψ ∼ N (µψ, σ2ψ), λ ∼ N (µλ, σ
2λ), and ψ ⊥⊥ λ.
We can compute the posterior distribution of ψ | Y and thereforethe problem seems to be solved: we learn on a non-identifiedparameter.
However, the posterior distribution of ψ is a function of theposterior distribution of the identified parameter because
E [ψ | Y ] = ηψ,λ +σ2ψ
σ2ψ + σ2λE [ψ + λ | Y ].
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 15
/ 33
Identifiability andthe Bayesian approach
Let Y | ψ, λ) ∼ N (ψ + λ, σ2Y ) be the statistical model(likelihood).
The parameters (ψ, λ) are not identified.
Prior distribution: ψ ∼ N (µψ, σ2ψ), λ ∼ N (µλ, σ
2λ), and ψ ⊥⊥ λ.
We can compute the posterior distribution of ψ | Y and thereforethe problem seems to be solved: we learn on a non-identifiedparameter.
However, the posterior distribution of ψ is a function of theposterior distribution of the identified parameter because
E [ψ | Y ] = ηψ,λ +σ2ψ
σ2ψ + σ2λE [ψ + λ | Y ].
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 15
/ 33
Identifiability andthe Bayesian approach
The previous analysis is an example of a general theorem.
The likelihood is exhaustively described by the identifiedparameter.
The learning-by-observing process is fully concentrated on theidentified parameter in the sense that, conditionally on it, there isnothing more to learn on the unidentified ones.
The identified parameter is the larger parameter that can berepresented as functions of the observations.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 16
/ 33
1PL model fixed effect I
The DGP corresponds to extract with replacement correct andincorrect answers from an urn like this:
It is assumed that {Ypi} are mutually independent and that
P(Ypi = 1) =εpηi.
The parameters of interest are identified provided that η1 = 1.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 17
/ 33
1PL model fixed effect I
The DGP corresponds to extract with replacement correct andincorrect answers from an urn like this:
It is assumed that {Ypi} are mutually independent and that
P(Ypi = 1) =εpηi.
The parameters of interest are identified provided that η1 = 1.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 17
/ 33
1PL model fixed effect I
The DGP corresponds to extract with replacement correct andincorrect answers from an urn like this:
It is assumed that {Ypi} are mutually independent and that
P(Ypi = 1) =εpηi.
The parameters of interest are identified provided that η1 = 1.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 17
/ 33
1PL model fixed effect II
The identification restriction η1 = 1 allows to define person abilityrelative to the standard item 1:
εp =P(Yp1 = 1)P(Yp1 = 0)
.
This leads to compare the ability of two persons w.r.t. the itemstandard 1 (Specific Objectivity):
εp > εq ⇐⇒ P(Yp1 = 1) > P(Yq1 = 1).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 18
/ 33
1PL model fixed effect II
The identification restriction η1 = 1 allows to define person abilityrelative to the standard item 1:
εp =P(Yp1 = 1)P(Yp1 = 0)
.
This leads to compare the ability of two persons w.r.t. the itemstandard 1 (Specific Objectivity):
εp > εq ⇐⇒ P(Yp1 = 1) > P(Yq1 = 1).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 18
/ 33
1PL model fixed effect III
The identification restriction leads to identify (and, therefore, todefine) the difficulty of an item j relative to the item standard 1:
ηi =P(Yp1 = 1)P(Yp1 = 0)
P(Ypi = 0)P(Ypi = 1)
,
ηi > ηj ⇐⇒ P(Ypj = 1) > P(Ypi = 1).
Joint representation “ability-difficulty”:
εp > ηi ⇐⇒ P(Ypi = 1) > P(Ypi = 0).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 19
/ 33
1PL model fixed effect III
The identification restriction leads to identify (and, therefore, todefine) the difficulty of an item j relative to the item standard 1:
ηi =P(Yp1 = 1)P(Yp1 = 0)
P(Ypi = 0)P(Ypi = 1)
,
ηi > ηj ⇐⇒ P(Ypj = 1) > P(Ypi = 1).
Joint representation “ability-difficulty”:
εp > ηi ⇐⇒ P(Ypi = 1) > P(Ypi = 0).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 19
/ 33
1PL model random effects I
Framework: the basic decompositions of Classical Test Theory,
Y = E (Y | θ) + [Y − E (Y | θ)],Var(Y ) = Var [E (Y | θ)] + E [Var(Y | θ)],
and the Axiom of Local Independence.
The DGP consists of choosing at random an urn. Once the urnhas been chosen, correct and incorrect responses are extractedwith replacement.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 20
/ 33
1PL model random effects I
Framework: the basic decompositions of Classical Test Theory,
Y = E (Y | θ) + [Y − E (Y | θ)],Var(Y ) = Var [E (Y | θ)] + E [Var(Y | θ)],
and the Axiom of Local Independence.
The DGP consists of choosing at random an urn. Once the urnhas been chosen, correct and incorrect responses are extractedwith replacement.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 20
/ 33
1PL model random effects II
ALI implies that responses of each person are correlated betweenthem.
Considering a semi-parametric version of the model (that is,θp
iid∼ G ), an identification restriction is needed, namely η1 = 1.
The identification restriction allows to identify (and, therefore, todefine), the difficulty of an item j :
ηi =P(Yp1 = 1,Ypj = 0)P(Yp1 = 0,Ypj = 1)
,
ηi > ηj ⇐⇒ P(Ypj = 1) > P(Ypi = 1).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 21
/ 33
1PL model random effects II
ALI implies that responses of each person are correlated betweenthem.
Considering a semi-parametric version of the model (that is,θp
iid∼ G ), an identification restriction is needed, namely η1 = 1.
The identification restriction allows to identify (and, therefore, todefine), the difficulty of an item j :
ηi =P(Yp1 = 1,Ypj = 0)P(Yp1 = 0,Ypj = 1)
,
ηi > ηj ⇐⇒ P(Ypj = 1) > P(Ypi = 1).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 21
/ 33
1PL model random effects II
ALI implies that responses of each person are correlated betweenthem.
Considering a semi-parametric version of the model (that is,θp
iid∼ G ), an identification restriction is needed, namely η1 = 1.
The identification restriction allows to identify (and, therefore, todefine), the difficulty of an item j :
ηi =P(Yp1 = 1,Ypj = 0)P(Yp1 = 0,Ypj = 1)
,
ηi > ηj ⇐⇒ P(Ypj = 1) > P(Ypi = 1).
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 21
/ 33
1PL model random effects III
What kind of scientific statements can be done with respect topersons? To what extent such statements relate persons anditems?
In the context of CTT, scientific statements at the person levelcan be done taking into account the dual decomposition
θp = E (θp | Yp) + [θp − E (θp | Yp)].
Given a responses pattern, which is the more probable urn fromwhich the responses come from? The conditional distribution of(θp | Yp) can be used to classify persons in “ability-urns”.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 22
/ 33
1PL model random effects III
What kind of scientific statements can be done with respect topersons? To what extent such statements relate persons anditems?In the context of CTT, scientific statements at the person levelcan be done taking into account the dual decomposition
θp = E (θp | Yp) + [θp − E (θp | Yp)].
Given a responses pattern, which is the more probable urn fromwhich the responses come from? The conditional distribution of(θp | Yp) can be used to classify persons in “ability-urns”.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 22
/ 33
1PL model random effects III
What kind of scientific statements can be done with respect topersons? To what extent such statements relate persons anditems?In the context of CTT, scientific statements at the person levelcan be done taking into account the dual decomposition
θp = E (θp | Yp) + [θp − E (θp | Yp)].
Given a responses pattern, which is the more probable urn fromwhich the responses come from? The conditional distribution of(θp | Yp) can be used to classify persons in “ability-urns”.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 22
/ 33
Summary
Rasch (1960) Lord (1952)
DGP Responses mutually independent For each person, responsesacross persons and items mutually dependent
Specific Valid InvalidObjectivity
Classification Invalid Valid
DifficultyP(Yp1 = 1)P(Yp1 = 0)
P(Ypi = 0)P(Ypi = 1)
P(Yp1 = 1,Ypj = 0)P(Yp1 = 0,Ypj = 1)
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 23
/ 33
1PL-G model random effects I
Axiom of Local Independence, θpiid∼ G ,
P(Ypi = 1) = ci + (1− ci )exp(θp − βi )
1+ exp(θp − βi ).
What means the parameters ci y βi?
The item parameters are identified provided that β1 = 0 y c1 = 0.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 24
/ 33
1PL-G model random effects I
Axiom of Local Independence, θpiid∼ G ,
P(Ypi = 1) = ci + (1− ci )exp(θp − βi )
1+ exp(θp − βi ).
What means the parameters ci y βi?
The item parameters are identified provided that β1 = 0 y c1 = 0.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 24
/ 33
1PL-G model random effects I
Axiom of Local Independence, θpiid∼ G ,
P(Ypi = 1) = ci + (1− ci )exp(θp − βi )
1+ exp(θp − βi ).
What means the parameters ci y βi?
The item parameters are identified provided that β1 = 0 y c1 = 0.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 24
/ 33
1PL-G model random effects II
Conditional odds with respect to the DGP:
Ok|j =P(Yk = 1 | Yj = 1, Yl = 0 para todo l 6= k, j)
P(Yk = 0 | Yj = 1, Yl = 0 para todo l 6= k, j).
Ok|0 =P(Yk = 1 | Yj = 0 para todo j 6= k)
P(Yk = 0 | Yl = 0 para todo l 6= k).
With respect to the standard item 1, it follows that
Ok|1 ≥ Ok|j ≥ Ok|0 for all k.
For the semiparametric 1PL model it follows that
Ok|1 = Ok|j ≥ Ok|0 for all k.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 25
/ 33
1PL-G model random effects II
Conditional odds with respect to the DGP:
Ok|j =P(Yk = 1 | Yj = 1, Yl = 0 para todo l 6= k, j)
P(Yk = 0 | Yj = 1, Yl = 0 para todo l 6= k, j).
Ok|0 =P(Yk = 1 | Yj = 0 para todo j 6= k)
P(Yk = 0 | Yl = 0 para todo l 6= k).
With respect to the standard item 1, it follows that
Ok|1 ≥ Ok|j ≥ Ok|0 for all k.
For the semiparametric 1PL model it follows that
Ok|1 = Ok|j ≥ Ok|0 for all k.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 25
/ 33
1PL-G model random effects II
Conditional odds with respect to the DGP:
Ok|j =P(Yk = 1 | Yj = 1, Yl = 0 para todo l 6= k, j)
P(Yk = 0 | Yj = 1, Yl = 0 para todo l 6= k, j).
Ok|0 =P(Yk = 1 | Yj = 0 para todo j 6= k)
P(Yk = 0 | Yl = 0 para todo l 6= k).
With respect to the standard item 1, it follows that
Ok|1 ≥ Ok|j ≥ Ok|0 for all k.
For the semiparametric 1PL model it follows that
Ok|1 = Ok|j ≥ Ok|0 for all k.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 25
/ 33
1PL-G model random effects III
A measure of the amount of information of item j w.r.t. the itemstandard 1 is defined as
IMj =Ok|j − Ok|0
Ok|1 − Ok|0∈ [0, 1].
It measure the distance between Ok|j and Ok|1 in(Ok|1 − Ok|0)-units.
IMj = 1 if and only if the behavior of the item is as a 1PL-item.
Therefore, IMj < 1 is equivlent to an item the behavior of whichis different from a 1PL-item.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 26
/ 33
1PL-G model random effects IV
If cj 6= 0 then
exp(βj) =
(1+ Oj |0
1−IMj
IMj
)(
Oj|0O1|0
)
βj is decreasing with respect to Oj |0: it seems “natural”.
βj is decreasing with respect to IMj : if the behavior of the item is“near” to a 1PL-item, then its value decreases.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 27
/ 33
1PL-G model random effects IV
If cj 6= 0 then
exp(βj) =
(1+ Oj |0
1−IMj
IMj
)(
Oj|0O1|0
)βj is decreasing with respect to Oj |0: it seems “natural”.
βj is decreasing with respect to IMj : if the behavior of the item is“near” to a 1PL-item, then its value decreases.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 27
/ 33
Modelo 1PL-G efecto aleatorio V
For an item k 6= 1,
ck =Ok|0(1− IMk)
1+ Ok|0(1− IMk)
ck is increasing with respect to Ok|0: if, conditionally to answerincorrectly all the items except the k-th item, is more probable toanswer correctly the item k than to answer it incorrectly, then ckincreases.
ck is decreasing with respecto to IMk : if the behavior of the itemis “near” to a 1PL-item, then ck decreases.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 28
/ 33
Modelo 1PL-G efecto aleatorio V
For an item k 6= 1,
ck =Ok|0(1− IMk)
1+ Ok|0(1− IMk)
ck is increasing with respect to Ok|0: if, conditionally to answerincorrectly all the items except the k-th item, is more probable toanswer correctly the item k than to answer it incorrectly, then ckincreases.
ck is decreasing with respecto to IMk : if the behavior of the itemis “near” to a 1PL-item, then ck decreases.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 28
/ 33
Summary
In a 1PL-G model, the parameters βj y cj represent how far thebehavior of the item is from a 1PL-item.
This comparison can be done because the identification restrictionensures to have a 1PL-type item in the model.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 29
/ 33
Discussion/Questions
The item parameters βj and cj can be written in terms of Oj |kand Oj |0.
These terms are defined conditionally on answering incorrectly allthe items, except one or answering incorrectly all the items.
Some conflict with the “definitions” of copying? The question ismeaningful because two examinees could copy incorrectalternative and therefore all their responses are incorrect. In sucha case, the parameters of the 1PL-G model seem to includecopying behavior: how is it possible to analyze copying behaviorusing IRT models?
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 30
/ 33
Discussion/Questions
Difficulty and guessing are not univocal designations: it is not welldone to make comparison between items parameters when fittingdifferent IRT models.
Aún queda por mejorar nuestra comprensión de lo que dichosmodelos permiten analizar.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 31
/ 33
Acknowledgement
Research supported by the FONDECYT Project Number 1181261of the Chilean government.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 32
/ 33
References
E. San Martín, J. González and F. Tuerlinckx (2009). Identified Parameters, Parameters of Interestand Their Relationships. Measurement: Interdisciplinary Research and Perspective 7, 95-103.
E. San Martín, A. Jara, J. -M. Rolin and M. Mouchart (2011). On the Bayesian nonparametricgeneralization of Rasch-type models. Psychometrika 76, 385-409.
E. San Martín and J. -M. Rolin (2013). Identification of Parametric Rasch-type Models. Journal ofStatistical Planning and Inference 143, 116-130.
E. San Martín, J.-M. Rolin and L. M. Castro (2013). Identification of the 1PL Model with guessingparameter: Parametric and Semi-parametric results. Psychometrika 78, 341-379.
E. San Martín (2015). Identification of Item Response Theory Models. In: W. van der Linden (Ed.),Handbook of Item Response Theory, Volumen 2, Chapter 8.
E. San Martín and P. De Boeck. (2015). What do you mean by a difficult item? On the interpretationof the difficulty parameter in Rasch models. In: Roger E. Millsap, Daniel M. Bolt, L. Andries van derArk, Wen-Chung Wang (Eds.), Quantitative Psychology Research, Springer Proceedings inMathematics & Statistics 89, Chapter 1.
E. San Martín, J. González and F. Tuerlinckx (2015). On the Unidentifiability of the Fixed-Effects3PL Model. Psychometrika 80, 450-467.
E. San Martín (2015). Modelos Rasch: ¿cuán (in-)coherentemente son presentados y utilizados?Actualidades en Psicología 29, 91-102.
P. Fariña, J. González and E. San Martín (2018). An identifiability-based strategy for theinterpretation of parameters in IRT models; under revision for Psychometrika.
E. San Martín (2018). Identifiability of Structural Characteristics: How relevant is in the BayesianApproach? Brazilian Journal of Probability and Statistics 32, 346-373.
Ernesto San Martín (LIES) Statistical meaning of IRT parametersAgosto 2018, Juiz de Fora, Brasil 33
/ 33